Construction machine, information processing device, program

Abstract

This invention provides a technique for determining the shape of the work object of a construction machine. In one embodiment of this invention, an excavator (100) includes a controller (30) and sensors (S1 to S6) that acquire data related to the track of the bucket (6) of the excavator (100). The controller (30) acquires data related to the track of the bucket (6) in response to the execution of the excavator's (100) digging or discharging action, and estimates the shape of the work object after the digging or discharging action is performed based on the acquired data.

Description

Technical Field

[0001] This invention relates to construction machinery, etc. Background Technology

[0002] Previously, construction machinery such as excavators that are operated by an operator or run autonomously was known (see Patent Document 1).

[0003] Previous technical documents Patent documents Patent Document 1: Japanese Patent Application Publication No. 2021-188432 Summary of the Invention

[0004] The technical problem to be solved by the invention For example, operators riding in excavators may encounter blind spots in the ground due to uneven sand or excavator attachments. Furthermore, when remotely operating an excavator, the operator needs to monitor the image from a camera mounted on the machine while operating it; however, based solely on the camera's image, it may be difficult to accurately determine the shape of the ground. Therefore, sometimes the shape of the work area is identified based on data from cameras or distance sensors, and displayed on a monitor in the operator's cab or on a remote control display.

[0005] Furthermore, for example, in the case of autonomous operation of an excavator, the track of the excavator's working part (bucket) is sometimes determined according to the shape of the sand and soil on the ground where the work is being carried out. In this case, the shape of the work object is identified based on data from cameras or range sensors, and the track of the construction machinery's working part is determined based on the shape of the work object.

[0006] However, when using a camera or range sensor mounted on an excavator, for example, the shape of a portion of the workpiece may be obscured by the excavator's attachments or the unevenness of the sand, preventing measurement. Therefore, for example, a hole may appear in the shape of the workpiece displayed on the screen, resulting in an undesirable situation from the operator's perspective. Furthermore, for example, because the shape of a portion of the workpiece cannot be measured, the accuracy related to the generation of the track for autonomous operation may deteriorate. Moreover, for example, without a camera or range sensor mounted on the excavator, the function of displaying the shape of the workpiece on the screen or the excavator's autonomous operation function is inherently unavailable.

[0007] Therefore, in view of the above issues, the aim is to provide a technology that can grasp the shape of the work object of construction machinery.

[0008] means for solving technical problems To achieve the above objectives, in one embodiment of the present invention, a construction machine is provided. The construction machinery is equipped with a processing device that acquires data representing the state of the construction machinery and related to changes in the shape of the work object in response to the actions of the construction machinery, and estimates the shape of the work object after the actions of the construction machinery based on the acquired data.

[0009] Furthermore, in another embodiment of the present invention, an information processing apparatus is provided. The information processing device acquires data representing the state of the construction machinery and related to changes in the shape of the work object in response to the actions of the construction machinery, and estimates the shape of the work object after the actions of the construction machinery based on the acquired data.

[0010] Furthermore, in another embodiment of the present invention, a program is provided. The program causes the information processing device to perform the following steps: The steps of acquiring data representing the state of the construction machinery in relation to changes in the shape of the work object being worked on, corresponding to the actions of the construction machinery; and The step of estimating the shape of the work object after the construction machinery has been operated based on the acquired data.

[0011] Furthermore, in another embodiment of the present invention, a program is provided. The procedure causes the support device to perform the following steps: A step of acquiring data representing the state of the construction machinery in relation to changes in the shape of the object being worked on, corresponding to the actions of the construction machinery; The steps of estimating the shape of the work object after the construction machinery has moved based on the acquired data; and The step of displaying the shape of the work object on the display unit based on the estimated shape of the work object.

[0012] Invention Effects According to the above implementation method, the shape of the object being worked on by the construction machinery can be determined. Attached Figure Description

[0013] Figure 1 This is a diagram illustrating an example of an operational support system.

[0014] Figure 2 This is a top view showing an example of an excavator.

[0015] Figure 3 This is a diagram illustrating an example of a structure related to the remote operation of an excavator.

[0016] Figure 4 This is a diagram illustrating an example of the hardware structure of an excavator.

[0017] Figure 5 This is a diagram illustrating an example of the hardware structure of an information processing device.

[0018] Figure 6 This is a functional block diagram illustrating the first example of the functional structure of an operational support system.

[0019] Figure 7 This is a functional block diagram illustrating the functional structure of the operation support system, as shown in the second example.

[0020] Figure 8 This is the third example of a functional block diagram illustrating the functional structure of an operational support system.

[0021] Figure 9 It is a diagram illustrating the method for estimating the shape of the work object.

[0022] Figure 10 This is a diagram illustrating an example of the measured and estimated data of the reaction force from the workpiece to the work area (bucket) during the digging action of an excavator.

[0023] Figure 11 This is another example of the measured and estimated data of the reaction force from the workpiece to the work area (bucket) during the digging action of an excavator.

[0024] Figure 12 This is another example of a diagram showing the measured and estimated data of the reaction force from the workpiece to the work area (bucket) during the digging action of an excavator.

[0025] Figure 13 This is a diagram illustrating an example of the relationship between the accuracy of data representing the shape of the work object and the predetermined actions of the excavator.

[0026] Figure 14 This is another example of a diagram showing the relationship between the accuracy of data representing the shape of the work object and the predetermined actions of the excavator.

[0027] Figure 15 This is a diagram illustrating the first example of processing related to the use of data representing the shape of a work object.

[0028] Figure 16 This is a second example of a process related to the use of data representing the shape of a work object.

[0029] Figure 17 This is the third example of a process related to the use of data representing the shape of a work object.

[0030] Figure 18 This is the fourth example of a diagram showing the processing related to the use of data representing the shape of a work object.

[0031] Figure 19 This is a flowchart illustrating, schematically, a first example of a process for obtaining data representing the shape of a work object.

[0032] Figure 20 This is a diagram showing an example of the area being observed.

[0033] Figure 21 This is a diagram showing an example of the affected area.

[0034] Figure 22 This is a flowchart illustrating, schematically, a second example of a process for obtaining data representing the shape of a work object.

[0035] Figure 23 This is a flowchart illustrating, schematically, a third example of a process for obtaining data representing the shape of a work object. Detailed Implementation

[0036] The embodiments will now be described with reference to the accompanying drawings.

[0037] [Overview of the Operation Support System] refer to Figures 1-3 An overview of the operation support system SYS involved in this embodiment will be provided.

[0038] Figure 1 This diagram illustrates an example of the operational support system SYS. Figure 1 The image shows a left view of the excavator 100. Figure 2 This is a top view showing an example of an excavator 100. Figure 3 This is a diagram illustrating an example of a structure related to the remote operation of the excavator 100. Hereinafter, the direction in which the accessory AT extends when viewed from above the excavator 100 will sometimes be indicated. Figure 2 The direction of the excavator 100 or the direction observed from the perspective of the excavator 100 is defined as "front" (the direction of the excavator 100).

[0039] like Figure 1 As shown, the operation support system SYS includes an excavator 100, an information processing device 200, and a sensor group 300.

[0040] The Operation Support System SYS uses the Information Processing Unit 200 to work with the Excavator 100 to provide support related to the operation of the Excavator 100.

[0041] The excavator 100 included in the operation support system SYS can be one or more.

[0042] In the Operation Support System SYS, Excavator 100 is a construction machine that is supported in relation to operation.

[0043] like Figure 1 , Figure 2 As shown, the excavator 100 includes a lower traveling body 1, an upper slewing body 3, an accessory AT including a boom 4, a stick 5 and a bucket 6, and a cab 10.

[0044] The lower traveling body 1 uses tracks 1C to move the excavator 100. Tracks 1C include a left track 1CL and a right track 1CR. Track 1CL is hydraulically driven by a travel hydraulic motor 1ML. Similarly, track 1CL is hydraulically driven by a travel hydraulic motor 1MR. Thus, the lower traveling body 1 is capable of self-propelled movement.

[0045] The upper rotating body 3 is rotatably mounted on the lower traveling body 1 via the rotating mechanism 2. For example, the upper rotating body 3 rotates relative to the lower traveling body 1 by hydraulically driving the rotating mechanism 2 via the rotating hydraulic motor 2M.

[0046] The boom 4 is mounted at the center of the front of the upper slewing body 3 in a manner that allows it to pitch around a rotation axis in the left-right direction. The stick 5 is mounted at the front end of the boom 4 in a manner that allows it to rotate around a rotation axis in the left-right direction. The bucket 6 is mounted at the front end of the stick 5 in a manner that allows it to rotate around a rotation axis in the left-right direction.

[0047] Bucket 6 is an example of an end-attachment, such as for excavation, ramp work, or ground clearing operations.

[0048] The bucket 6 is mounted on the front end of the boom 5 in a manner that allows for appropriate replacement to correspond to the work performed by the excavator 100. That is, a different type of bucket, such as a relatively large bucket, a slope bucket, or a dredging bucket, can be installed at the front end of the boom 5 instead of the bucket 6. Furthermore, end attachments other than buckets, such as mixers, hydraulic breakers, or shredders, can also be installed at the front end of the boom 5. Additionally, optional attachments such as quick-connect couplings or tilting rotators can be provided between the boom 5 and the end attachments.

[0049] The boom 4, stick 5, and bucket 6 are hydraulically driven by the boom cylinder 7, stick cylinder 8, and bucket cylinder 9, respectively.

[0050] The cab 10 is a control room for the operator to sit in and operate the excavator 100. The cab 10 is, for example, mounted on the front left side of the upper rotating body 3.

[0051] The excavator 100 is equipped with a communication device 60, which enables it to communicate with the information processing device 200 via a predetermined communication line NW.

[0052] Communication lines (NW) can include, for example, local area networks (LANs) at construction sites. Furthermore, communication lines (NW) can also include wide area networks (WANs). Wide area networks include, for example, mobile communication networks with base stations as endpoints, satellite communication networks utilizing communication satellites, and the Internet. Additionally, communication lines (NW) can also include, for example, short-range communication lines based on wireless communication standards such as WiFi or Bluetooth (registered trademark).

[0053] For example, the excavator 100 causes the driven components such as the lower walking body 1 (i.e., the left and right pairs of tracks 1CL and 1CR), the upper slewing body 3, the boom 4, the stick 5, and the bucket 6 to move in response to the operation of the operator sitting in the cab 10.

[0054] Furthermore, the excavator 100 can be configured to be remotely operated from outside the excavator 100 (remote operation) instead of being operated by an operator sitting in the cab 10, or, in addition to being operable by an operator sitting in the cab 10, it can also be remotely operated from outside the excavator 100. In the case of remote operation of the excavator 100, the cab 10 can be unmanned. Furthermore, if the excavator 100 is dedicated to remote operation, the cab 10 can be omitted. Hereinafter, the description will assume that the operator's operation includes at least one of the following: operation of the operating device 26 by the operator in the cab 10 and remote operation by an external operator.

[0055] For example, such as Figure 3 As shown, remote operation includes operating the excavator 100 via operation inputs related to the actuator of the excavator 100, made by a remote operation support device 400. This remote operation support device 400 can communicate with the excavator 100 via a communication line NW. The remote operation support device 400 can be separate from or integrated with the information processing device 200.

[0056] The remote operation support device 400 may be installed, for example, in a management center that manages the operation of the excavator 100 from the outside. Furthermore, the remote operation support device 400 may be a portable operating terminal, in which case the operator can remotely operate the excavator 100 while directly monitoring its operating status from its vicinity.

[0057] For example, the excavator 100 can transmit an image (hereinafter referred to as "peripheral image") showing the state of its surroundings, including the front of the excavator 100, to the remote operation support device 400 via the communication device 60 described later. This image is generated based on a camera image output by a camera device mounted on the excavator 100. Furthermore, the excavator 100 can also transmit a camera image output by the camera device to the remote operation support device 400 via the communication device 60. The remote operation support device 400 processes the camera image received from the excavator 100 to generate the peripheral image. The remote operation support device 400 can then display the peripheral image showing the state of its surroundings, including the front of the excavator 100, on its own display device. Additionally, various information images (information screens) displayed on the output device 50 (display device 50A) inside the excavator 100's cab 10 can also be displayed on the display device of the remote operation support device 400. Therefore, the operator using the remote operation support device 400 can remotely operate the excavator 100 while checking the images or information displayed on the display device showing the status of the excavator 100's surroundings. Then, the excavator 100 can activate its actuators to drive driven components such as the lower walking body 1, upper slewing body 3, boom 4, stick 5, and bucket 6, based on the remote operation signal received by the communication device 60 from the remote operation support device 400 indicating the content of the remote operation.

[0058] Furthermore, remote operation may include methods such as operating the excavator 100 based on external voice or gesture input from people around the excavator 100 (e.g., workers). Specifically, the excavator 100 recognizes voice or gestures from nearby workers using a voice input device (e.g., a microphone) or gesture input device (e.g., a camera). Then, the excavator 100 can activate actuators based on the recognized voice or gestures to drive driven components such as the lower walking body 1 (left and right tracks 1C), upper slewing body 3, boom 4, stick 5, and bucket 6.

[0059] Furthermore, the excavator 100 can also automatically operate the actuators regardless of the operator's actions. Thus, the excavator 100 can achieve the function of automatically operating at least a portion of the driven components such as the lower traveling body 1, the upper rotating body 3, and the attachment AT, i.e., the so-called "automatic operation function" or "machine control (MC) function".

[0060] Automatic operation functions may include, for example, semi-automatic operation functions (operation support type MC functions). Semi-automatic operation functions are functions that automatically activate driven components (actuators) other than the driven components (actuators) of the workpiece in response to operator input. Furthermore, automatic operation functions may also include fully automatic operation functions (fully automatic type MC functions). Fully automatic operation functions are functions that automatically activate at least a portion of multiple driven components (hydraulic actuators) without operator input. In the excavator 100, when the fully automatic operation function is active, the cab 10 can be unmanned. Furthermore, if the excavator 100 is dedicated to fully automatic operation, the cab 10 may be omitted. Moreover, semi-automatic or fully automatic operation functions may include, for example, rule-based automatic operation functions. Rule-based automatic operation functions are automatic operation functions that automatically determine the action content of the driven components (actuators) of the workpiece according to pre-defined rules. Furthermore, semi-automatic or fully automatic operation functions may also include autonomous operation functions. The autonomous operation function is an automatic operation function in the following way: the excavator 100 autonomously makes various judgments and determines the action content of the driven components (hydraulic actuators) of the automatic operation object based on the judgment results.

[0061] Furthermore, the operation of the excavator 100 can be remotely monitored. In this case, a remote monitoring support device with the same functions as the remote operation support device 400 can be installed. The remote monitoring support device is, for example, an information processing device 200. Thus, the monitor, as the user of the remote monitoring support device, can monitor the operating status of the excavator 100 while checking the surrounding image displayed on the display device of the remote monitoring support device. Furthermore, for example, if it is deemed necessary from a safety point of view, the monitor can intervene in the operator's operation of the excavator 100 or its automatic operation by making predetermined inputs through the input device of the remote monitoring support device, thereby causing the excavator 100 to stop urgently.

[0062] The information processing device 200 cooperates with the excavator 100 through communication to provide support related to the operation of the excavator 100.

[0063] The information processing device 200 may be a server device or a management terminal device, such as a management office located at the construction site of the excavator 100 or a management center located at a different location from the construction site of the excavator 100, for managing the operation status of the excavator 100. The server device may be a local server, a cloud server, or an edge server. The management terminal device may be a fixed terminal device such as a desktop PC (Personal Computer), or a portable terminal device (mobile terminal) such as a tablet, smartphone, or laptop. In the latter case, construction site workers, supervisors, and managers can carry the portable information processing device 200 around the construction site. Furthermore, in the latter case, the operator may bring the portable information processing device 200 into the cab of the excavator 100.

[0064] The information processing device 200 acquires data related to the operating status from the excavator 100, for example. Thus, the information processing device 200 can grasp the operating status of the excavator 100 and monitor for any abnormalities. Furthermore, the information processing device 200 can, for example, display the data related to the operating status of the excavator 100 via the display device 208 (described later) and allow the user to confirm. Additionally, the information processing device 200 can, for example, enable a learning model to learn the operating status of the excavator 100 and generate a completed learning model to support the operation of the excavator 100.

[0065] Furthermore, the information processing device 200 can send various data, such as programs or reference data used in the processing of the controller 30, to the excavator 100. As a result, the excavator 100 can use the various data downloaded from the information processing device 200 to perform various processes related to the operation of the excavator 100.

[0066] Sensor group 300 is installed at the construction site of excavator 100.

[0067] For example, when the operation support system SYS includes multiple excavators 100, a sensor group 300 is provided for each excavator 100. Furthermore, when the multiple excavators 100 included in the operation support system SYS are operating at the same construction site, a single sensor group 300 can be shared by the multiple excavators 100.

[0068] Sensor group 300 includes sensors 300-1 to 300-M (M: an integer greater than 2). Sensors 300-1 to 300-M measure the state of objects at the construction site surrounding the excavator 100 and acquire measurement data related to that state. The objects at the construction site include the work object of the excavator 100. The work object is, for example, the sand in the work area surrounding the excavator 100. Furthermore, the objects at the construction site, in addition to the work object (the sand in the work area) surrounding the excavator 100, also include, for example, other construction machinery such as excavators and bulldozers, or work vehicles such as sand transport trucks in the vicinity of the excavator 100. The state of the objects includes their shape or characteristics.

[0069] Sensors 300-1 to 300-M include, for example, range sensors (distance sensors). Range sensors may include, for example, LIDAR (Light Detecting and Ranging), millimeter-wave radar, ultrasonic sensors, and infrared sensors. Furthermore, sensors 300-1 to 300-M may include, for example, stereo cameras, TOF (Time of Flight) cameras, and other 3D cameras capable of acquiring data related to distance (depth) in addition to two-dimensional images. Moreover, range sensors and 3D cameras may be combined in sensors 300-1 to 300-M. Thus, sensor group 300 can acquire measurement data representing the shape of objects at the construction site surrounding excavator 100. Hereinafter, for convenience, sensors capable of acquiring measurement data representing the shape of objects, such as range sensors or 3D cameras, are sometimes referred to as "shape sensors."

[0070] Furthermore, sensors 300-1 to 300-M may include multi-wavelength beam splitters. Multi-wavelength beam splitters may include, for example, multispectral cameras or hyperspectral cameras. Thus, sensor group 300, for example, can acquire measurement data representing the characteristics of objects at the construction site surrounding the excavator 100, such as the hardness or moisture content of sand. Hereinafter, for convenience, sensors capable of acquiring measurement data representing the characteristics of objects, such as multi-wavelength beam splitters, will sometimes be referred to as "characteristic sensors."

[0071] For example, sensors 300-1 to 300-M include multiple shape sensors. Furthermore, these multiple shape sensors can be positioned at different locations within the construction site surrounding the excavator 100, and their respective sensing ranges can overlap with the sensing range of at least one other shape sensor. Thus, for example, even if occlusion prevents the acquisition of measurement data representing the shape of a portion of the object within the sensing range from being obtained from the measurement data of one shape sensor, measurement data representing the shape of the object within that range can still be obtained from other shape sensors. Therefore, sensor group 300 can more reliably acquire measurement data representing the shape of objects at the construction site surrounding the excavator 100.

[0072] Furthermore, sensors 300-1 to 300-M may also include multiple characteristic sensors. Moreover, these multiple characteristic sensors can be positioned at different locations within the construction site surrounding the excavator 100, and their respective sensing ranges can overlap with the sensing range of at least one other characteristic sensor. Thus, for example, even if occlusion prevents the acquisition of measurement data representing the characteristics of a portion of the object within the sensing range from being obtained from the measurement data of one characteristic sensor, measurement data representing the characteristics of the object within that range can still be obtained from other characteristic sensors. Therefore, the sensor group 300 can more reliably acquire measurement data representing the characteristics of objects at the construction site surrounding the excavator 100.

[0073] Furthermore, sensors 300-1 to 300-M may also include sensors that have the functions of both shape sensors and characteristic sensors (hereinafter referred to as "integrated sensors"). In this case, sensors 300-1 to 300-M may include multiple integrated sensors. Moreover, the multiple characteristic sensors may be set at different locations on the construction site around the excavator 100, and each of them may be configured such that its sensing range overlaps with the sensing range of at least one other characteristic sensor.

[0074] Alternatively, the sensor group 300 can simply include only a shape sensor or a characteristic sensor. Furthermore, the operation support system SYS can replace the sensor group 300 by simply including only a single sensor capable of acquiring measurement data related to the state of objects at the construction site surrounding the excavator 100.

[0075] Sensors 300-1 to 300-M can be fixed to the construction site surrounding the excavator 100, or they can be mounted on a mobile object that can move within the construction site surrounding the excavator 100. The mobile object includes, for example, construction machinery or work vehicles that move within the construction site. Furthermore, mobile objects that can move within the construction site can include, for example, drones or other flying objects that fly over the construction site.

[0076] The outputs (measurement data) of sensors 300-1 to 300-M are input to the information processing device 200 via communication line NW. The outputs of sensors 300-1 to 300-M can be directly input to the information processing device 200 via communication line NW, for example. Alternatively, the outputs of sensors 300-1 to 300-M can be temporarily input to the excavator 100 via communication line NW, and then input to the information processing device 200 via the excavator 100. Furthermore, when sensors 300-1 to 300-M are mounted on a predetermined device such as the aforementioned mobile body, the outputs of sensors 300-1 to 300-M can be temporarily input into the predetermined device and then input to the information processing device 200 from that device.

[0077] [Hardware structure of the operation support system] Besides reference Figures 1-3 In addition, also refer to Figure 4 , Figure 5 The hardware structure of the SYS operating support system will be described.

[0078] Furthermore, the hardware structure of the remote operation support device 400 can be the same as that of the information processing device 200. Therefore, illustrations and descriptions related to the hardware structure of the remote operation support device 400 are omitted.

[0079] <Hardware Structure of Excavators> Figure 4 This is a diagram illustrating an example of the hardware structure of the excavator 100.

[0080] In addition, Figure 4 In the diagram, double lines represent the path for transmitting mechanical power, solid lines represent the path for the flow of high-pressure working oil driving the hydraulic actuator, dashed lines represent the path for transmitting pilot pressure, and dotted lines represent the path for transmitting electrical signals.

[0081] The excavator 100 includes components such as a hydraulic drive system related to the hydraulic drive of the driven component, an operating system related to the operation of the driven component, a user interface system related to information exchange with the user, a communication system related to communication with the outside world, and a control system related to various controls.

[0082] Hydraulic Drive System like Figure 4 As shown above, the hydraulic drive system of the excavator 100 includes a hydraulic actuator HA, which hydraulically drives the lower traveling body 1 (left and right tracks 1C), upper slewing body 3, boom 4, stick 5, and bucket 6, respectively. Furthermore, the hydraulic drive system of the excavator 100 according to this embodiment includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

[0083] The hydraulic actuator HA includes the travel hydraulic motors 1ML and 1MR, the swing hydraulic motor 2M, the boom cylinder 7, the stick cylinder 8, and the bucket cylinder 9, etc.

[0084] Alternatively, the excavator 100 may replace part or all of the hydraulic actuator HA with an electric actuator. That is, the excavator 100 may be a hybrid excavator or an electric excavator.

[0085] Engine 11 is the prime mover of excavator 100 and the main power source in the hydraulic drive system. Engine 11 is, for example, a diesel engine that uses diesel fuel. Engine 11 is, for example, mounted at the rear of the upper rotating body 3. Engine 11 rotates at a constant target speed under the direct or indirect control of controller 30 (described later) to drive the main pump 14 and pilot pump 15.

[0086] Alternatively, other prime movers (e.g., electric motors) can be mounted on the excavator 100 to replace the engine 11, or other prime movers besides the engine 11 can be mounted on the excavator 100.

[0087] The regulator 13 controls (adjusts) the output of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swashplate of the main pump 14 (hereinafter referred to as "deflection angle") according to the control command from the controller 30.

[0088] The main pump 14 supplies working oil to the control valve 17 via a high-pressure hydraulic line. Similar to the engine 11, the main pump 14 is, for example, mounted at the rear of the upper rotating body 3. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable-capacity hydraulic pump, and as described above, under the control of the controller 30, the piston stroke length is adjusted by adjusting the deflection angle of the swashplate via the regulator 13, thereby controlling the discharge flow rate or discharge pressure.

[0089] Control valve 17 drives hydraulic actuators HA according to operator input to operating device 26, remote operation commands, or operation instructions corresponding to automatic operation functions. Control valve 17 is, for example, mounted in the central part of the upper rotating body 3. As described above, control valve 17 is connected to main pump 14 via high-pressure hydraulic lines and selectively supplies working oil from main pump 14 to each hydraulic actuator according to operator input or operation instructions corresponding to automatic operation functions. Specifically, control valve 17 includes multiple control valves (directional valves) that control the flow rate and direction of working oil supplied from main pump 14 to each hydraulic actuator HA.

[0090] "operating system" like Figure 4 As shown, the operating system of the excavator 100 includes a pilot pump 15, an operating device 26, a hydraulic control valve 31, a shuttle valve 32, and a hydraulic control valve 33.

[0091] Pilot pump 15 supplies pilot pressure to various hydraulic devices via pilot line 25. Similar to engine 11, pilot pump 15 is, for example, mounted at the rear of upper rotating body 3. Pilot pump 15 is, for example, a fixed-capacity hydraulic pump, as described above, driven by engine 11.

[0092] Alternatively, the pilot pump 15 can be omitted. In this case, the relatively high-pressure working oil discharged from the main pump 14, after being reduced to a relatively low pressure by a predetermined pressure reducing valve, can be supplied as pilot pressure to various hydraulic equipment.

[0093] The operating device 26 is located near the operator's seat in the cab 10 and is used by the operator to operate various driven components. Specifically, the operating device 26 is used by the operator to operate the hydraulic actuators HA that drive each driven component, thereby enabling the operator to operate the driven component that is driven by the hydraulic actuators HA. The operating device 26 includes a pedal device or a lever device for operating each driven component (hydraulic actuator HA).

[0094] For example, such as Figure 4 As shown, the operating device 26 is hydraulically piloted. Specifically, the operating device 26 utilizes working oil supplied from the pilot pump 15 via pilot line 25 and its branch pilot line 25A to output pilot pressure corresponding to the operation to pilot line 27A on the secondary side. Pilot line 27A is connected to one inlet port of shuttle valve 32 and is connected to control valve 17 via pilot line 27 connected to the outlet port of shuttle valve 32. Thus, pilot pressure corresponding to the operation related to various driven components (hydraulic actuators HA) in the operating device 26 can be input to control valve 17 via shuttle valve 32. Therefore, control valve 17 can drive each hydraulic actuator HA according to the operation of the operating device 26 by the operator or others.

[0095] Furthermore, the operating device 26 can also be electrically powered. In this case, the pilot line 27A, shuttle valve 32, and hydraulic control valve 33 are omitted. Specifically, the operating device 26 outputs an electrical signal (hereinafter referred to as the "operation signal") corresponding to the operation content, and the operation signal is input to the controller 30. Then, the controller 30 outputs a control command corresponding to the content of the operation signal to the hydraulic control valve 31, that is, a control signal corresponding to the operation content of the operating device 26. As a result, a pilot pressure corresponding to the operation content of the operating device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26.

[0096] Furthermore, the control valve (directional valve) built into the control valve 17 that drives each hydraulic actuator HA can also be a solenoid type. In this case, the operating signal output from the operating device 26 can be directly input to the control valve 17 (i.e., a solenoid control valve).

[0097] Furthermore, as described above, some or all of the hydraulic actuator HA can be replaced with an electric actuator. In this case, the controller 30 can output control commands corresponding to the operation content of the operating device 26 or the remote operation content specified by the remote operation signal to the electric actuator or the driver that drives the electric actuator. Moreover, in the case of remotely operating the excavator 100, the operating device 26 can be omitted.

[0098] Hydraulic control valves 31 are provided for each driven component (hydraulic actuator HA) of the operating device 26, and for each driving direction of the driven component (hydraulic actuator HA) (e.g., the lifting and lowering direction of the boom 4). For example, two hydraulic control valves 31 are provided for each double-acting hydraulic actuator HA used to drive the lower traveling body 1, upper slewing body 3, boom 4, stick 5, and bucket 6. The hydraulic control valves 31 may be provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17, and configured to change their flow area (i.e., the cross-sectional area through which the working oil can flow). Thus, the hydraulic control valves 31 can output a predetermined pilot pressure to the secondary pilot line 27B using the working oil supplied to the pilot pump 15 through the pilot line 25B. Therefore, the hydraulic control valves 31 can indirectly act on the control valve 17 with a predetermined pilot pressure corresponding to the control signal from the controller 30 through the shuttle valve 32 between the pilot line 27B and the pilot line 27. Therefore, for example, the controller 30 can supply pilot pressure from the hydraulic control valve 31 to the control valve 17 corresponding to the operation command corresponding to the automatic operation function, thereby realizing the operation of the excavator 100 based on the automatic operation function.

[0099] Furthermore, the controller 30 can also control the hydraulic control valve 31 to achieve remote operation of the excavator 100. Specifically, the controller 30 outputs a control signal to the hydraulic control valve 31 via the communication device 60, corresponding to the content of the remote operation specified by the remote operation signal received from the remote operation support device 400. Thus, the controller 30 can supply pilot pressure from the hydraulic control valve 31 to the control valve 17 corresponding to the content of the remote operation, thereby realizing the operation of the excavator 100 based on the operator's remote operation.

[0100] Furthermore, when the operating device 26 is electric, the controller 30 can directly supply pilot pressure from the hydraulic control valve 31 to the control valve 17 corresponding to the operation content (operation signal) of the operating device 26, thereby realizing the operation of the excavator 100 based on the operator's operation.

[0101] The shuttle valve 32 has two inlet ports and one outlet port, outputting working oil with the higher pilot pressure of the pilot pressure input to the two inlet ports to the outlet port. Similar to the hydraulic control valve 31, the shuttle valve 32 is provided for each driven element (hydraulic actuator HA) of the operating device 26, and for each drive direction of the driven element (hydraulic actuator HA). For example, two shuttle valves 32 are provided for each double-acting hydraulic actuator HA used to drive the lower traveling body 1, upper slewing body 3, boom 4, stick 5, and bucket 6, etc. One of the two inlet ports of the shuttle valve 32 is connected to the pilot line 27A on the secondary side of the operating device 26 (specifically, the aforementioned lever or pedal device included in the operating device 26), and the other is connected to the pilot line 27B on the secondary side of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected to the pilot port of the corresponding control valve in the control valve 17 via the pilot line 27. The corresponding control valve refers to the control valve of the hydraulic actuator HA, which drives the lever or pedal device connected to one inlet port of the shuttle valve 32. Therefore, these shuttle valves 32 can apply the higher of the pilot pressure from the pilot line 27A on the secondary side of the operating device 26 and the pilot pressure from the pilot line 27B on the secondary side of the hydraulic control valve 31 to the pilot port of the corresponding control valve. That is, the controller 30 can control the corresponding control valve independently of the operator's operation of the operating device 26 by outputting a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26 from the hydraulic control valve 31. Thus, the controller 30 can control the movement of the driven components (lower traveling body 1, upper slewing body 3, boom 4, stick 5, and bucket 6) independently of the operator's operating state of the operating device 26, thereby achieving automatic operation or remote operation functions.

[0102] A hydraulic control valve 33 is provided in the pilot line 27A connecting the operating device 26 and the shuttle valve 32. The hydraulic control valve 33 is configured, for example, to change its flow area. The hydraulic control valve 33 operates in response to a control signal input from the controller 30. Therefore, when the operating device 26 is being operated by the operator, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26. Thus, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator HA corresponding to the operation of the operating device 26. Furthermore, for example, even when the operating device 26 is being operated, the controller 30 can reduce the pilot pressure output from the operating device 26 to a level lower than the pilot pressure output from the hydraulic control valve 31. Therefore, by controlling the hydraulic control valves 31 and 33, the controller 30 can reliably apply the desired pilot pressure to the pilot port of the control valve within the control valve 17, regardless of the operation of the operating device 26. Thus, the controller 30 can, for example, more accurately realize the automatic operation function or remote operation function of the excavator 100 by controlling the hydraulic control valve 33 in addition to controlling the hydraulic control valve 31.

[0103] User Interface Systems like Figure 4 As shown, the user interface system of the excavator 100 includes an operating device 26, an output device 50, and an input device 52.

[0104] The output device 50 outputs various information to users of the excavator 100 (e.g., the operator of the cab 10 or an external remote operator) or people around the excavator 100 (e.g., workers or drivers of work vehicles).

[0105] For example, output device 50 includes lighting equipment or display device 50A that visually outputs various information (see reference). Figure 7 Lighting equipment includes, for example, warning lights (indicator lights). Display devices 50A include, for example, liquid crystal displays (LCDs) or organic EL (Electroluminescence) displays. For example, such as... Figure 2 As shown, the lighting equipment or display device 50A can be installed inside the cab 10 and output various information to the operator inside the cab 10 in a visual manner. Furthermore, the lighting equipment or display device 50A can also be installed on the side of the upper rotating body 3, for example, and output various information to workers around the excavator 100 in a visual manner.

[0106] Furthermore, the output device 50 may also include a sound output device that outputs various information in an auditory manner. The sound output device may include, for example, a buzzer or a loudspeaker. The sound output device may be installed at least one inside or outside the cab 10, and output various information audibly to the operator inside the cab 10 or to people (workers, etc.) around the excavator 100.

[0107] Furthermore, the output device 50 may also include a device that outputs various information in a tactile manner, such as by vibration of the operator's seat.

[0108] Input device 52 receives various inputs from the user of excavator 100, and the signals corresponding to the received inputs are input to controller 30. For example, such as Figure 2 As shown, the input device 52 is installed inside the cab 10 and receives input from the operator inside the cab 10. Alternatively, the input device 52 may be installed on the side of the upper rotating body 3 and receive input from personnel around the excavator 100.

[0109] For example, input device 52 includes an operation input device that accepts mechanical operation-based input from a user. The operation input device may include a touch panel mounted on display device 50A, a touchpad disposed around display device 50A, a push-button switch, a lever, a toggle switch, a rotary switch disposed on operation device 26 (lever device), etc.

[0110] Furthermore, the input device 52 may also include a voice input device that accepts voice input from the user. The voice input device may include, for example, a microphone.

[0111] Furthermore, the input device 52 may also include a gesture input device that accepts user gesture input. For example, the gesture input device may include a camera device that captures the state of the user's gestures.

[0112] Furthermore, the input device 52 may also include a biometric input device for accepting biometric input from a user. Biometric input may include, for example, the input of biometric information such as the user's fingerprint or iris scan.

[0113] Communication Systems like Figure 4 As shown, the communication system of the excavator 100 involved in this embodiment includes a communication device 60.

[0114] The communication device 60 is connected to an external communication line NW and communicates with a device separately installed from the excavator 100. In addition to the device located outside the excavator 100, the device separately installed from the excavator 100 may also include a portable terminal device (mobile terminal) brought into the cab 10 by the user of the excavator 100. The communication device 60 may, for example, include devices compliant with 4G (4G... th Generation: Fourth Generation Mobile Communication) or 5G (5G) th The communication device 60 may include standard mobile communication modules such as fifth-generation mobile communication (5G). Furthermore, the communication device 60 may also include, for example, a satellite communication module. Additionally, the communication device 60 may also include, for example, a WiFi communication module or a Bluetooth (registered trademark) communication module. Furthermore, in the presence of multiple connectable communication lines NW, the communication device 60 may include multiple communication devices depending on the type of communication line NW.

[0115] For example, communication device 60 communicates with external devices such as information processing device 200 or remote operation support device 400 within the construction site via local communication lines constructed at the construction site. These local communication lines may be, for example, mobile communication lines based on local 5G (so-called local 5G) constructed at the construction site or local area networks based on WiFi 6.

[0116] Furthermore, the communication device 60 can also communicate with external information processing device 200 or remote operation support device 400 located at the construction site via a wide area network, including a wide area communication line at the construction site.

[0117] Control Systems like Figure 4 As shown, the control system of the excavator 100 includes a controller 30. Furthermore, the control system of the excavator 100 according to this embodiment includes an operating pressure sensor 29, a sensor 40, and sensors S1 to S9.

[0118] The controller 30 performs various controls related to the excavator 100.

[0119] The functionality of controller 30 can be implemented using any hardware or any combination of hardware and software. For example, ... Figure 3 As shown, the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C, and an interface device 30D connected via a bus BS1.

[0120] Auxiliary storage device 30A is a non-volatile storage unit that stores the program to be installed, as well as necessary files or data. Auxiliary storage device 30A may be, for example, EEPROM (Electrically Erasable Programmable Read-Only Memory) or flash memory.

[0121] For example, when a program start instruction is present, memory device 30B loads the program from auxiliary storage device 30A so that CPU 30C can read the program. Memory device 30B is, for example, SRAM (Static Random Access Memory).

[0122] CPU 30C, for example, executes a program loaded into memory device 30B and implements various functions of controller 30 according to the program's commands.

[0123] The interface device 30D functions, for example, as a communication interface for connecting to communication lines inside the excavator 100. The interface device 30D may also include multiple different types of communication interfaces depending on the type of communication line to be connected.

[0124] Furthermore, the interface device 30D functions as an external interface for reading data from or writing data to the storage medium. The storage medium can be, for example, a special tool connected to a connector located inside the cab 10 via a detachable cable. The storage medium can also be a common storage medium such as an SD memory card or a USB (Universal Serial Bus) memory. Thus, programs implementing various functions of the controller 30 can be provided, for example, by a portable storage medium and installed in the auxiliary storage device 30A of the controller 30. Furthermore, the program can also be downloaded from another computer (e.g., information processing device 200) outside the excavator 100 and installed in the auxiliary storage device 30A via the communication device 60.

[0125] In addition, some of the functions of controller 30 can also be implemented by other controllers (control devices). That is, the functions of controller 30 can be implemented in a distributed manner by multiple controllers mounted on excavator 100.

[0126] The operating pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot-operated device 26, that is, the pilot pressure corresponding to the operating state of each driven component (hydraulic actuator) in the operating device 26. The detection signal of the pilot pressure corresponding to the operating state of each driven component (hydraulic actuator HA) in the operating device 26 detected by the operating pressure sensor 29 is input to the controller 30.

[0127] Furthermore, when the operating device 26 is electrically powered, the operating pressure sensor 29 is omitted. This is because the controller 30 can grasp the operating status of each driven component operated by the operating device 26 based on the operating signals input from the operating device 26.

[0128] Sensor 40, for example, acquires measurement data related to the shape of objects around excavator 100.

[0129] For example, sensor 40 is a shape sensor, such as a range sensor or a 3D camera, capable of acquiring measurement data representing the shape of objects around the excavator 100. Furthermore, sensor 40 may also be an integrated sensor that, in addition to having the functions of a shape sensor, also has the functions of a characteristic sensor, such as a multi-wavelength beam splitter camera, capable of acquiring measurement data representing the characteristics of objects around the excavator 100.

[0130] For example, such as Figure 2 As shown, sensor 40 includes sensors 40F, 40B, 40L, and 40R. Sensor 40F measures the state (shape or characteristics) of objects in front of the upper rotating body 3. Sensor 40B measures the state of objects behind the upper rotating body 3. Sensor 40L measures the state of objects to the left of the upper rotating body 3. Sensor 40R measures the state of objects to the right of the upper rotating body 3. Thus, sensor 40 can measure the state of objects in all directions, i.e., a 360-degree angular range centered on the excavator 100, when viewed from above. Hereinafter, sensors 40F, 40B, 40L, and 40R will sometimes be collectively referred to or individually as "sensor 40X".

[0131] The output data of sensor 40 (sensor 40X) (i.e., measurement data related to the state of objects around excavator 100) is input to controller 30 via a one-to-one communication line or vehicle network. Thus, for example, controller 30 can determine the state of objects around excavator 100, such as their shape or characteristics, based on the output data of sensor 40X.

[0132] Alternatively, some or all of the sensors 40B, 40L, and 40R can be omitted.

[0133] Sensor S1 is mounted on boom 4 and measures the posture state of boom 4. Sensor S1 outputs measurement data representing the posture state of boom 4. The posture state of boom 4 is, for example, the posture angle (hereinafter referred to as "boom angle") of the base end of the connection between boom 4 and upper rotating body 3 about the rotation axis. Sensor S1 may include, for example, a rotary potentiometer, rotary encoder, accelerometer, angular accelerometer, six-axis sensor, IMU (Inertial Measurement Unit), etc. The same applies to sensors S2 to S4. Furthermore, sensor S1 may include a cylinder sensor that detects the extension and retraction position of boom cylinder 7. The same applies to sensors S2 and S3. The output of sensor S1 (measurement data representing the posture state of boom 4) is input to controller 30. Thus, controller 30 can grasp the posture state of boom 4.

[0134] Sensor S2 is mounted on the boom 5 and measures the attitude state of the boom 5. Sensor S2 outputs measurement data representing the attitude state of the boom 5. The attitude state of the boom 5 is, for example, the attitude angle (hereinafter referred to as "boom angle") of the base end of the boom 5 connected to the boom 4 about the axis of rotation. The output of sensor S2 (the measurement data representing the attitude state of the boom 5) is input to controller 30. Thus, controller 30 can grasp the attitude state of the boom 5.

[0135] Sensor S3 is mounted on bucket 6 and measures the posture of bucket 6. Sensor S3 outputs measurement data representing the posture of bucket 6. The posture of bucket 6 is, for example, the posture angle (hereinafter referred to as "bucket angle") of the base end of the connection between bucket 6 and stick 5 about the axis of rotation. The output of sensor S3 (the measurement data representing the posture of bucket 6) is input to controller 30. Thus, controller 30 can grasp the posture of bucket 6.

[0136] Sensor S4 measures the posture of the excavator 100's body (e.g., the upper rotating body 3). Sensor S4 outputs measurement data representing the posture of the excavator 100's body. The posture of the excavator 100's body is, for example, its tilt relative to a predetermined reference plane (e.g., a horizontal plane). For example, sensor S4 is mounted on the upper rotating body 3 and measures the tilt angles (hereinafter referred to as "forward tilt angle" and "left-right tilt angle") of the excavator 100 about two axes in the forward and backward directions. The output of sensor S4 (the measurement data representing the posture of the excavator 100's body) is input to controller 30. Thus, controller 30 can grasp the posture (tilt state) of the body (upper rotating body 3).

[0137] Sensor S5 is mounted on the upper rotating body 3 and measures the rotation state of the upper rotating body 3. Sensor S5 outputs measurement data indicating the rotation state of the upper rotating body 3. Sensor S5, for example, measures the rotational angular velocity or rotational angle of the upper rotating body 3. Sensor S5 may include, for example, a gyroscope sensor, a resolver, a rotary encoder, etc. The output of sensor S5 (measurement data indicating the rotation state of the upper rotating body 3) is input to controller 30. Thus, controller 30 can grasp the rotation state of the upper rotating body 3, such as the rotational angle.

[0138] The controller 30 can determine (estimate) the position of the front end (bucket 6) of the attachment AT based on the output of the sensors S1 to S5.

[0139] Alternatively, if sensor S4 includes a gyroscope sensor, a six-axis sensor, an IMU, or the like capable of detecting angular velocities around three axes, the rotational state (e.g., rotational angular velocity) of the upper rotating body 3 can be detected based on the detection signal from sensor S4. In this case, sensor S5 can be omitted.

[0140] Sensor S6 measures the position of excavator 100. Sensor S6 can measure the position in world (global) coordinates or in local coordinates at the construction site. In the former case, sensor S6 is, for example, a GNSS (Global Navigation Satellite System) sensor. In the latter case, sensor S6 is a transceiver capable of communicating with a device serving as a reference for the position at the construction site and outputting a signal corresponding to the position relative to the reference. The output of sensor S6 is input to controller 30.

[0141] Sensor S7 measures the pressure (cylinder pressure) of the oil chamber of boom cylinder 7. Sensor S7 may include, for example, a sensor that measures the cylinder pressure (rod pressure) of the rod-side oil chamber of boom cylinder 7 and a sensor that measures the cylinder pressure (bottom pressure) of the bottom-side oil chamber. The output of sensor S7 (the measured cylinder pressure data of boom cylinder 7) is input to controller 30.

[0142] Sensor S8 measures the pressure (cylinder pressure) of the oil chamber of the boom cylinder 8. Sensor S8 may include, for example, a sensor that measures the cylinder pressure (rod pressure) of the rod-side oil chamber of the boom cylinder 8 and a sensor that measures the cylinder pressure (bottom pressure) of the bottom-side oil chamber of the boom cylinder 8. The output of sensor S8 (the measured cylinder pressure data of the boom cylinder 8) is input to controller 30.

[0143] Sensor S9 measures the pressure (cylinder pressure) of the oil chamber of bucket cylinder 9. Sensor S9 may include, for example, a sensor that measures the cylinder pressure (rod pressure) of the rod-side oil chamber of bucket cylinder 9 and a sensor that measures the cylinder pressure (bottom pressure) of the bottom-side oil chamber of bucket cylinder 9. The output of sensor S9 (the measurement data of the cylinder pressure of bucket cylinder 9) is input to controller 30.

[0144] The controller 30 can grasp the load status acting on the attachment AT based on the output of the sensors S7 to S9. The load acting on the attachment AT includes, for example, the reaction force acting on the bucket 6 from the work object (sand on the ground) or the weight of the sand contained in the bucket 6.

[0145] In addition to sensors S1 to S9, the excavator 100 may also be equipped with other sensors capable of monitoring its status. For example, the excavator 100 may have an orientation sensor capable of detecting its own orientation. An orientation sensor could be, for example, an electronic compass including a geomagnetic sensor.

[0146] <Hardware Structure of Information Processing Device> Figure 5 This is a diagram illustrating an example of the hardware structure of the information processing device 200.

[0147] The functions of the information processing device 200 are implemented through any hardware or any combination of hardware and software. For example, such as Figure 5 As shown, the information processing device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a high-speed computing device 205, a communication interface 206, an input device 207, a display device 208, and a sound output device 209. They are connected via a bus BS2.

[0148] External interface 201 functions as an interface for reading data from or writing data to storage medium 201A. Storage medium 201A may include, for example, floppy disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), BDs (Blu-ray Discs), SD cards, and USB storage devices. Thus, information processing device 200 can read various types of data used in processing through storage medium 201A and store them in auxiliary storage device 202, or install programs that implement various functions.

[0149] In addition, the information processing device 200 can also obtain various data or programs used in the processing from external devices via the communication interface 206.

[0150] Auxiliary storage device 202 stores various installed programs and files or data required for various processes. Auxiliary storage device 202 may include, for example, HDD (Hard Disc Drive), SSD (Solid State Disk), flash memory, etc.

[0151] When a program start instruction is present, memory device 203 reads the program from auxiliary storage device 202 and stores it. Memory device 203 may include, for example, DRAM (Dynamic Random Access Memory) or SRAM.

[0152] CPU 204 executes various programs loaded from auxiliary storage device 202 into memory device 203, and performs various functions related to information processing device 200 according to the programs.

[0153] The high-speed computing device 205 works in conjunction with the CPU 204 to perform computational processing at a relatively high speed. The high-speed computing device 205 may include, for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or a FPGA (Field-Programmable Gate Array).

[0154] In addition, depending on the required processing speed, the high-speed computing unit 205 can be omitted.

[0155] The communication interface 206 serves as an interface for communication connection with external devices. Thus, the information processing device 200 can communicate with external devices such as the excavator 100 via the communication interface 206. Furthermore, the communication interface 206 can have multiple types of communication interfaces depending on the communication method with the device to be connected.

[0156] The input device 207 accepts various inputs from the user. The input device 207 includes a remote operation device for remotely operating the excavator 100.

[0157] Input device 207 may include, for example, an input device that accepts mechanical operation input from a user (hereinafter referred to as "operation input device"). Remote operation devices may be operation input devices. Operation input devices may include, for example, buttons, toggle switches, levers, keyboards, mice, touch panels mounted on display device 208, touchpads disposed separately from display device 208, etc.

[0158] Furthermore, the input device 207 may also include a voice input device capable of accepting voice input from a user. The voice input device may include, for example, a microphone capable of collecting the user's voice.

[0159] Furthermore, the input device 207 may also include a gesture input device capable of accepting gesture input from the user. The gesture input device may include, for example, a camera capable of capturing the state of the user's gestures.

[0160] Furthermore, the input device 207 may also include a biometric input device capable of accepting biometric input from a user. For example, a biometric input device may include a camera capable of acquiring image data containing information related to the user's fingerprint or iris.

[0161] Display device 208 displays information screens or operation screens to the user of information processing device 200. Display device 208 is, for example, a liquid crystal display or an organic EL (Electroluminescence) display.

[0162] The sound output device 209 uses sound to transmit various information to the user of the information processing device 200. The sound output device 209 is, for example, a buzzer, an alarm, a speaker, etc.

[0163] [Functional Structure of the Operation Support System] Besides reference Figures 1-5 In addition, also refer to Figures 6 to 14 The functional structure of the Operation Support System (SYS) will be explained. Specifically, the functional structure of the Operation Support System (SYS) related to the generation of the track of the working part of the excavator 100 will be explained.

[0164] <Example 1> Figure 6 This is a functional block diagram illustrating the first example of the functional structure of the SYS (System for Operation Support).

[0165] Hereinafter, "the track of the working part of the excavator 100" will be used to mean both the path (i.e., trajectory) that the working part of the excavator 100 has already moved and the path that it may move in the future. The working part is equivalent to the front end of the attachment AT used to apply changes to the work object. Specifically, the working part is the bucket 6.

[0166] The excavator 100 includes a support device 150. In this example, the support device 150 provides support related to the operation of the excavator 100, which operates through an autonomous operation function.

[0167] like Figure 6 As shown, the support device 150 includes a controller 30, a hydraulic control valve 31, a sensor 40, and sensors S1 to S9.

[0168] The controller 30 includes an action log providing unit 301 and an operation support unit 302 as functional units.

[0169] Furthermore, when the operation support system SYS includes multiple excavators 100, it is possible for controller 30 to include only the excavator 100 in the former of the action log providing unit 301 and the operation support unit 302, or for controller 30 to include only the excavator 100 in the latter. In this case, the excavator 100 in the former only has the function of providing operation support for the excavator 100 in the latter, namely, acquiring the action log of the excavator 100 and providing it to the information processing device 200. The same applies to the second and third examples described later.

[0170] The information processing device 200 includes a log acquisition unit 2001, a simulator unit 2002, a log storage unit 2003, a training data generation unit 2004, a machine learning unit 2005, a learned model storage unit 2006, and a distribution unit 2007 as functional units.

[0171] The action log providing unit 301 is a functional unit for acquiring the action log of the excavator 100 when it performs a predetermined action and providing it to the information processing device 200.

[0172] Pre-defined actions include, for example, those used during excavation operations such as digging, boom lifting and slewing, boom lowering and slewing, soil removal, and sweeping. Furthermore, pre-defined actions can also include, during land preparation operations, such as digging, soil removal, sweeping, horizontal traction, compaction, and sweeping. Additionally, pre-defined actions can also include, during slope operations, such as cutting and compaction. A sweeping action, for example, involves using attachment AT to push the bucket 6 forward along the ground surface, sweeping sand and soil forward with the back of the bucket 6. During the sweeping action, for example, attachment AT lowers the boom 4 and opens the stick 5. A horizontal traction action, for example, involves using attachment AT to move the tips of the bucket 6 teeth in a roughly horizontal, forward-facing manner to level the unevenness of the ground (terrain surface). During the horizontal traction action, for example, attachment AT raises the boom 4 and closes the stick 5. The compaction action is, for example, the action of pressing the ground with the back of the bucket 6 by operating the attachment AT. Furthermore, the compaction action can also be the action of pressing the ground by moving the bucket 6 up and down while simultaneously striking the ground with the back of the bucket 6. Additionally, the compaction action can also be the action of pushing the bucket 6 forward along the ground surface, sweeping sand and soil to a predetermined position with the back of the bucket 6, and then pressing the ground at that predetermined position with the back of the bucket 6. In the compaction action, for example, the attachment AT lowers the boom 4 while pressing the ground. The sweeping action is, for example, the action of rotating the upper slewing body 3 to rotate the bucket 6 left and right along the ground. Furthermore, the sweeping action can also be, for example, the action of alternating left and right rotations of the bucket 6 while simultaneously pushing the bucket 6 forward by operating the attachment AT and the upper slewing body 3. In the sweeping action, for example, the upper slewing body 3 alternately and repeatedly rotates left and right. Furthermore, during the sweeping action, in addition to alternating left and right rotation of the upper rotating body 3, the attachment AT can also perform the lowering action of the boom 4 and the opening action of the stick 5, similar to the sweeping action.

[0173] The operation log of the excavator 100 is timing data representing the operational status of the excavator 100. For example, the operation log of the excavator 100 includes timing data representing the operator's actions. This timing data representing the operator's actions may be, for example, the timing output data of the operating pressure sensor 29 corresponding to the hydraulic pilot-operated device 26, or the timing output data (operation signal data) of the operating device 26 corresponding to the electric operating device 26. Furthermore, the operation log of the excavator 100 may be the timing output data of sensors S1 to S5, or timing data representing the posture status of the excavator 100 obtained from the output data of sensors S1 to S5.

[0174] For example, the action log providing unit 301 acquires the action log of an operator with a long driving history and relatively rich experience (hereinafter, for convenience, referred to as a "skilled operator") operating the excavator 100, and provides it to the information processing device 200. Thus, as described later, a fully learned model LM2 capable of reproducing the actions of the excavator 100 based on the action log of the excavator 100 can be generated using machine learning based on the skilled operator's operation.

[0175] Furthermore, as described later, the learned model LM2 can be omitted, and only the learned model LM1 can be generated. In this case, the operation log of the excavator 100 provided to the information processing device 200 may include the operation log of the excavator 100 operated by an operator other than a skilled operator, and may also include the operation log corresponding to the operation of the excavator 100 based on the automatic operation function. Furthermore, the operation log of the excavator 100 for the learned model LM1 and the operation log of the excavator 100 for the learned model LM2 can be obtained separately.

[0176] The action log providing unit 301 includes an action log recording unit 301A, an action log storage unit 301B, and an action log sending unit 301C.

[0177] The action log recording unit 301A acquires the action log of the excavator 100 when it performs a predetermined action and records it in the action log storage unit 301B. For example, each time the excavator 100 performs a predetermined action, the action log recording unit 301A records the action log of that action in the action log storage unit 301B.

[0178] The action log of the excavator 100 is stored in the action log storage unit 301B. For example, for each predetermined action performed by the excavator 100, the action log is associated with the data of the time (date and time) at which the predetermined action is performed and stored in the action log storage unit 301B. The data of the time at which the predetermined action is performed includes data of the time when the excavator 100 starts the predetermined action and the time when the predetermined action ends. Furthermore, when multiple predetermined actions are specified, for each predetermined action performed by the excavator 100, the action log, the data of the time at which the predetermined action is performed, and the identification information of the predetermined action performed are associated and stored in the action log storage unit 301B. Hereinafter, for convenience, the data associated with the action log of the excavator 100 is sometimes referred to as "ancillary data". For example, in the action log storage unit 301B, for each predetermined action performed by the excavator 100, record data representing the correspondence between the action log and the ancillary data is accumulated, thereby constructing a database of the action log of the excavator 100 when performing the predetermined action.

[0179] Alternatively, the action logs that have been sent to the information processing device 200 by the action log sending unit 301C (described later) can be deleted afterward.

[0180] The action log sending unit 301C sends the action logs stored in the action log storage unit 301B, which record the action logs of the excavator 100 when performing predetermined actions, along with the associated data, to the information processing unit 200 via the communication device 60. Furthermore, the action log sending unit 301C can also send recorded data showing the correspondence between the action logs and associated data of the excavator 100 for each predetermined action performed by the excavator 100 to the information processing unit 200.

[0181] For example, the action log sending unit 301C, upon receiving a request from the information processing device 200 to send the unsent action logs of the excavator 100 stored in the action log storage unit 301B, along with associated data, to the information processing device 200. Furthermore, the action log sending unit 301C can also automatically send the unsent action logs of the excavator 100 stored in the action log storage unit 301B, along with associated data, to the information processing device 200 at a predetermined time. The predetermined time could be, for example, when the excavator 100 stops operating (key switch is "off") or starts operating (key switch is "on").

[0182] The log acquisition unit 2001 acquires logs of the excavator 100 when it performs a predetermined action.

[0183] The log for the excavator 100 performing the predetermined action includes an action log for the excavator 100 performing the predetermined action and a status log for the work object. The status log for the work object includes data indicating the status of the work object before and after the excavator 100 performs the predetermined action. The status of the work object includes its shape (i.e., the terrain shape of the ground area) or the characteristics of the sand and soil. The action log for the excavator 100 performing the predetermined action is uploaded from the excavator 100. The status log for the work object when the excavator 100 performs the predetermined action is obtained based on measurement data uploaded from the sensor group 300 and supplementary data (data on the moment the predetermined action is performed) uploaded from the excavator 100.

[0184] The simulator 2002 uses a virtual model of the excavator 100 and the work object (sand) to perform computer simulations related to the predetermined actions of the excavator 100.

[0185] For example, the discrete element method (DEM) is used to model the sand and soil of the work object as a collection of tiny particles. Therefore, the simulator unit 2002 can virtually reproduce the overall behavior of the work object sand and soil as a collection, or the reaction forces from the sand and soil, by having the virtual model of the excavator 100 perform predetermined actions such as digging, and by analyzing the movement of each tiny particle.

[0186] The simulator unit 2002 acquires data on the track of the working part of the excavator 100 and data on the state of the work object (sand) before and after the execution of the predetermined action as a log when the excavator 100 performs the predetermined action through computer simulation. The former data is equivalent to the action log when the excavator 100 performs the predetermined action through computer simulation, and the latter data is equivalent to the state log of the work object when the excavator 100 performs the predetermined action through computer simulation.

[0187] The simulator unit 2002 uses the states of various work objects (sand) and the tracks of various working parts of the excavator 100 to perform computer simulations of multiple modes related to the predetermined actions of the excavator 100. As a result, the simulator unit 2002 can accumulate logs of the excavator 100 performing predetermined actions under different conditions in the log storage unit 2003.

[0188] The log storage unit 2003 stores, in an accumulation manner, the logs acquired by the log acquisition unit 2001 and the simulator unit 2002 during the excavator 100's execution of predetermined actions. For example, the log storage unit 2003 stores, in an associative manner, the action logs of each predetermined action actually performed by the excavator 100 or performed through computer simulation, the status logs of the work object, and related data. In the log storage unit 2003, the logs acquired by the log acquisition unit 2001 and the logs acquired by the simulator unit 2002 can be stored in a identifiable manner or in a mixed manner without being identifiable.

[0189] The training data generation unit 2004 generates training data for machine learning based on logs stored in the log storage unit 2003 during the excavator 100's execution of predetermined actions, and outputs a collection of multiple training data sets, i.e., a training dataset. The training data generation unit 2004 can automatically generate training data through batch processing, or it can generate training data based on user input from the information processing device 200. The training data generation unit 2004 includes training data generation units 2004A and 2004B.

[0190] The training data generation unit 2004A generates a training dataset for generating the learned model LM1. The learned model LM1 infers the shape of the work object after the excavator 100 performs a predetermined action based on the predetermined input data. The shape of the work object refers to, for example, the terrain shape of the work object's ground surface, and more specifically, the shape of the undulations and other sand and soil exposed on the work object's ground surface.

[0191] The input data corresponding to the learned model LM1 includes data representing the state of the work object before the excavator 100 performs the predetermined action and data on the trajectory of the working part when the excavator 100 performs the predetermined action. The state of the work object includes, for example, the shape of the work object. Furthermore, the state of the work object may also include the characteristics of the sand or soil. For example, data representing the characteristics of the sand or soil includes data on the angle of repose of the sand or soil. Therefore, the learned model LM1 can consider the angle of repose of the sand or soil to infer a more accurate shape of the work object. Furthermore, the input data corresponding to the learned model LM1 may include data representing the digging reaction force when the excavator 100 performs the predetermined action. The digging reaction force refers to the reaction force acting from the ground on the working part of the excavator 100 (specifically, the bucket 6). Therefore, for example, even if the bucket 6 comes into contact with underground rock or the like, resulting in the inability to dig sand or soil, the learned model LM1 can still consider the digging reaction force to estimate an accurate shape of the sand or soil.

[0192] The training data is a combination of input data for the categories specified for the learned model LM1 and data representing the correct inference results corresponding to that input data (correct data). Correct data is data representing the state of the work object after the excavator 100 performs a predetermined action, corresponding to the input data included in the training data. Furthermore, when multiple categories of predetermined actions are specified, the learned model LM1 can be generated for each category of predetermined actions. In this case, the training data generation unit 2004A generates a training dataset for each category of predetermined actions.

[0193] The training dataset for generating the learned model LM1 can be generated, for example, from the logs acquired by the log acquisition unit 2001 and the logs output from the simulator unit 2002. Alternatively, the training dataset for generating the learned model LM1 can be generated solely from the logs acquired by the log acquisition unit 2001 and the logs output from the simulator unit 2002. In this case, the simulator unit 2002 can be omitted. Furthermore, the training dataset for generating the learned model LM1 can also be generated solely from the logs acquired by the log acquisition unit 2001 and the logs output from the simulator unit 2002. In this case, the motion log providing unit 301 of the sensor group 300 and the excavator 100 can be omitted. Additionally, the training dataset for generating the learned model LM1 can include a basic training dataset and a final adjustment (micro-call) training dataset. In this case, since the basic training dataset requires a large amount of data, it can be generated based on the logs output from the simulator unit 2002, and the final adjustment training dataset can be generated based on the logs acquired by the log acquisition unit 2001. The following content regarding the generation method of these learned models LM1 can also be applied to the generation method of learned models LM2 to LM4 described later, and can also be used as a reference for the generation method of learned models LM2 to LM4.

[0194] The training data generation unit 2004B generates training data for generating the learned model LM2. The learned model LM2 infers the target trajectory of the working part under the predetermined action of the excavator 100 based on the predetermined input data.

[0195] The input data corresponding to the learned model LM2 includes, for example, data indicating the state of the work object surrounding the excavator 100. As described above, the data indicating the state of the work object includes, for example, data indicating the shape of the work object. Furthermore, as described above, the data indicating the state of the work object may also include, for example, data indicating the characteristics of the sand in the work object. Additionally, the input data corresponding to the learned model LM2 may also include data indicating the target shape of the work object. Furthermore, when the predetermined action of the excavator 100 is a soil discharge action, the input data corresponding to the learned model LM2 may include data indicating the weight or volume of the sand contained in the bucket 6 before soil discharge.

[0196] The training data is a combination of input data for the types specified by the learned model LM2 and data representing the correct inference results corresponding to the input data (correct data). Specifically, the input data included in the training data includes data representing the state of the work object before the excavator 100 performs the predetermined action. Furthermore, the input data included in the training data may also include data representing the target shape corresponding to the work object. Furthermore, if the predetermined action of the excavator 100 is a dumping action, the data included in the training data may include input data representing the weight or volume of sand contained in the bucket 6 before dumping. Furthermore, the correct data included in the training data includes data representing the trajectory of the work part when the excavator 100 performs the predetermined action based on the state of the work object corresponding to the input data and through the operation of a skilled operator. That is, the training data generation unit 2004B generates a training dataset based on the logs obtained by the log acquisition unit 2001 when the excavator 100 performs the predetermined action through the operation of a skilled operator. Furthermore, when multiple types of predetermined actions are specified, a learned model LM2 can be generated for each type of predetermined action. In this case, the training data generation unit 2004B generates a training dataset for each type of predetermined action.

[0197] The Machine Learning Division 2005 generated fully learned models LM1 and LM2 by performing machine learning on the basic learning model based on the training dataset generated by the Training Data Generation Division 2004. The fully learned models (basic learning models) include neural networks such as DNNs (Deep Neural Networks).

[0198] The Machine Learning Division 2005 includes Machine Learning Division 2005A and 2005B.

[0199] The machine learning unit 2005A performs machine learning on the basic learning model M1 based on the training dataset output from the training data generation unit 2004A. As a result, the machine learning unit 2005A can generate a fully learned model LM1, which takes as input data the state of the work object before the excavator 100 performs a predetermined action and data on the track of the work area when the excavator 100 performs the predetermined action, and outputs (infers) the state of the work object after the excavator 100 performs the predetermined action. For example, the machine learning unit 2005A can optimize the learning model M1 using an error backpropagation algorithm based on the error between the output data of the learning model M1 and the correct data for the input data included in the training data, and generate the fully learned model LM1. The generation of the fully learned models LM2 to LM4, described later, can be done in the same way. Furthermore, the machine learning unit 2005A can perform additional learning on the fully learned model LM1 to correct the error between the inference result based on the fully learned model LM1 and the actual measurement result of the sensor 40. In this case, data based on the inference results of the learned model LM1 and the actual measurement results of the sensor 40 are uploaded from the excavator 100 to the information processing device 200 for further learning.

[0200] The machine learning unit 2005B performs machine learning on the basic learning model M2 based on the training dataset output from the training data generation unit 2004B. As a result, the machine learning unit 2005B can generate a learned model LM2, which can take the state data of the working objects around the excavator 100 as input and output (infer) the target trajectory of the working part under the predetermined action of the excavator 100.

[0201] Furthermore, the machine learning unit 2005B can also generate the learned model LM2 by applying reinforcement learning to the learning model M2 instead of supervised learning. In this case, the training data generation unit 2004B is omitted. For example, the machine learning unit 2005B performs machine learning on the learning model M1 based on the logs obtained by the log acquisition unit 2001 and the simulator unit 2002 to maximize the predetermined reward related to job efficiency. At this time, by linking with the simulator unit 2002 and having the simulator unit 2002 test a large number of action modes, machine learning on the learning model M1 can be performed more efficiently.

[0202] The learned models LM1 and LM2, output by the machine learning unit 2005, are stored in the learned model storage unit 2006. Furthermore, when the learned model LM1 is relearned or supplemented through the machine learning unit 2005A, the learned model LM1 in the learned model storage unit 2006 is updated. The same applies when the learned model LM2 is relearned or supplemented through the machine learning unit 2005B. Moreover, when the learned model LM1 is updated, the previous learned model LM1 can be saved in the learned model storage unit 2006 or another storage unit in a way that allows for reuse. The same applies when the learned model LM2 is updated. Therefore, for example, if there is a problem with the updated learned model LM1 or learned model LM2, the previous learned model LM1 or learned model LM2 can be recovered and reused.

[0203] The distribution department 2007 distributed the learned data of models LM1 and LM2 to excavator 100.

[0204] For example, if a learned model LM1 is generated or updated by the machine learning unit 2005A, the distribution unit 2007 distributes the most recently generated or updated learned model LM1 to the excavator 100. Furthermore, the distribution unit 2007 can distribute the latest learned model LM1 from the learned model storage unit 2006 to the excavator 100 based on a signal received from the excavator 100 requesting the distribution of the learned model LM1. The same applies to the learned model LM2.

[0205] The work support unit 302 is a functional unit used to provide work support for the excavator 100 that operates through the autonomous operation function.

[0206] The operation support unit 302 includes a learning completed model storage unit 302A, an operation object shape acquisition unit 302B, a target trajectory generation unit 302C, and a motion control unit 302D.

[0207] The learned models LM1 and LM2, which are distributed from the information processing device 200 and received through the communication device 60, are stored in the learned model storage unit 302A.

[0208] The object shape acquisition unit 302B acquires data representing the shape of the object being worked on by the excavator 100 based on the trajectory (track) of the working part when the excavator 100 performs a predetermined action. Specifically, the object shape acquisition unit 302B can acquire data representing the shape of the object being worked on by estimating the change in terrain shape (shape of sand) from before the predetermined action is performed, based on the trajectory of the working part when the excavator 100 performs the predetermined action. Furthermore, the object shape acquisition unit 302B can also acquire data representing the shape of the object being worked on by considering characteristic measurement data of the sand of the object being worked on measured by the sensor 40 (characteristic sensor) or measurement data from sensors S7 to S9 (i.e., data related to the reaction force from the sand of the object being worked on to the working part).

[0209] At this time, the object shape acquisition unit 302B can, based on the measurement data of the object shape measured by the sensor 40, interpolate the shape of the sand at locations that cannot be measured by the sensor 40 according to the trajectory of the working part when the excavator 100 performs a predetermined action. Furthermore, the object shape acquisition unit 302B can also acquire data representing the shape of the object without using the sensor 40, based on the initial state of the object shape of the excavator 100, and according to the trajectory of the working part each time the excavator 100 performs a predetermined action, while estimating the change in the shape of the sand. In this case, the sensor 40 can be omitted. The initial state of the object shape of the excavator 100 can be distributed from outside the excavator 100 via the communication device 60 or acquired by the user via the input device 52. Furthermore, the initial state (initial shape) of the object shape of the excavator 100 can be fixed, for example, as a plane at the same height as the ground where the excavator 100's tracks are installed.

[0210] For example, the object shape acquisition unit 302B estimates the sand shape after the excavator 100 performs the predetermined action based on the sand shape before the excavator 100 performs the predetermined action and the trajectory of the working part when the excavator 100 performs the predetermined action, using a learned model LM1. Then, the object shape acquisition unit 302B can acquire data representing the current state of the object by integrating the above estimation result with measurement data of the sand shape after the excavator 100 performs the predetermined action, measured by the sensor 40. Specifically, the object shape acquisition unit 302B performs data integration by interpolating the data from the above estimation result to areas in the observation area around the excavator 100 where the sensor 40 cannot measure the sand shape. The observation area refers to the range around the excavator 100 where the object shape acquisition unit 302B acquires data representing the shape of the object being worked on by the excavator 100. Therefore, even if the sensor 40 is unable to acquire measurement data of a part of the work object in the observation area due to obstruction, the controller 30 can still acquire data representing the shape of the work object in the observation area, including the shape of the work object at that location.

[0211] The shape of the work object before the excavator 100 performs the predetermined action is, for example, equivalent to the previous output of the work object shape acquisition unit 302B. The trajectory of the work part in the predetermined action of the excavator 100 is acquired, for example, based on the output of sensors S1 to S6.

[0212] Furthermore, based on the output of sensor 40, if obstacles exist within the observation area surrounding the excavator 100, the object shape acquisition unit 302B can exclude the area containing the obstacle from the estimation of the object's shape. Obstacles include, for example, moving objects such as construction machinery or work vehicles, utility poles, or fences. Moreover, when multiple excavators 100 are operating at the same construction site, they can share data representing the trajectory of their work area during the execution of a predetermined action. Therefore, the object shape acquisition unit 302B (learned model LM1) can also consider the changes in the object's shape caused by the predetermined actions of other excavators 100. Thus, the object shape acquisition unit 302B (learned model LM1) can more accurately estimate the shape of the object surrounding the excavator 100. Part or all of the function of the object shape acquisition unit 302B can also be delegated to an external device with relatively high processing power (e.g., information processing device 200). Therefore, even when its own processing power is insufficient, the controller 30 can still consider the trajectory of the working part when the excavator 100 performs the predetermined action and obtain data representing the shape of the working object after the excavator 100 performs the predetermined action.

[0213] The target track generation unit 302C generates the target track in the predetermined action of the excavator 100 based on the estimation result of the object shape acquisition unit 302B (the state of the object after the excavator 100 performs the predetermined action).

[0214] For example, the target track generation unit 302C generates the target track of the working part in the predetermined action of the excavator 100 using the learned model LM2 based on the shape of the working object obtained by the working object shape acquisition unit 302B.

[0215] Alternatively, the target trajectory generation unit 302C can replace the learned model LM2 by applying any known method to generate a target trajectory for the working part of the excavator 100 that matches the state (prediction result) of the work objects surrounding the excavator 100. In this case, the training data generation unit 2004C and the machine learning unit 2005B can be omitted. For example, the target trajectory generation unit 302C can generate target trajectory data for the working part of the excavator 100 based on the shape of the work object obtained by the work object shape acquisition unit 302B using MPC (Model Predictive Control). Furthermore, the target trajectory generation unit 302C can also generate target trajectory data for the working part of the excavator 100 by optimizing a predefined reference trajectory for the working part of the excavator 100 based on the shape of the work object obtained by the work object shape acquisition unit 302B.

[0216] The motion control unit 302D causes the excavator 100 to perform a predetermined action, so that a predetermined part of the excavator 100 moves along a target track generated by the target track generation unit 302C. Specifically, while the motion control unit 302D knows the position of the working part based on the outputs of sensors S1 to S5, it controls the hydraulic control valve 31 to cause the excavator 100 to perform a predetermined action, so that the working part of the excavator 100 moves along the target track. Thus, the excavator 100 can autonomously advance while performing a predetermined action along the shape of the work object.

[0217] Thus, in this example, the controller 30 acquires data representing the state of the excavator 100 and related to the shape of the work object in accordance with a predetermined action of the excavator 100. Then, the controller 30 estimates the shape of the work object (ground) of the excavator 100 based on the acquired data. Specifically, the controller 30 acquires data representing the trajectory of the working part when the excavator 100 performs the predetermined action, and estimates the shape of the work object (ground) after the excavator 100 performs the predetermined action based on this data. Therefore, for example, even in situations where the sensor 40 cannot measure the location of the sand shape due to obstruction, the controller 30 can consider changes in the sand shape caused by the movement of the working part of the excavator 100 and interpolate the sand shape at that location. Furthermore, even without using the sensor 40, the shape of the work object of the excavator 100 can be estimated based on a historical record of changes in the sand shape caused by the movement of the working part of the excavator 100. Therefore, the controller 30 can more accurately grasp the shape of the sand and soil of the excavator 100's work object in the observation area, and as a result, can more accurately generate the target trajectory of the excavator 100's work area. Thus, the controller 30 enables the excavator 100 to perform autonomous operation more accurately.

[0218] Furthermore, some or all of the functions of the object shape acquisition unit 302B, the target track generation unit 302C, and the motion control unit 302D can be transferred to the information processing device 200. Therefore, the processing load on the excavator 100 can be reduced for processing related to the generation of the target track at the working part of the excavator 100 or for processing related to the control of the excavator 100's motion. The same applies to the third example described later.

[0219] <Example 2> Figure 7 This is the second example of a functional block diagram illustrating the functional structure of the SYS operating support system.

[0220] Hereinafter, the same reference numerals will be used to mark the same structures as those in the first example above, and the parts that are different from those in the first example above will be explained in detail.

[0221] Similar to the first example above, the excavator 100 includes a support device 150. In this example, the support device 150 provides support to the user who operates the excavator 100 to perform work or monitors the operation of the excavator 100.

[0222] like Figure 7 As shown, in this example, the support device 150 includes a controller 30, a sensor 40, a display device 50A, and sensors S1 to S9. Furthermore, in the case of remote operation of the excavator 100, the support device 150 may include a communication device 60.

[0223] Similar to the first example above, the controller 30 includes an action log providing unit 301 and an operation support unit 302 as functional units.

[0224] Similar to the first example above, the action log providing unit 301 includes an action log recording unit 301A, an action log storage unit 301B, and an action log sending unit 301C.

[0225] Similar to the first example above, the information processing device 200 includes a log acquisition unit 2001, a simulator unit 2002, a log storage unit 2003, a training data generation unit 2004, a machine learning unit 2005, a learned model storage unit 2006, and a distribution unit 2007 as functional units.

[0226] In this example, the information processing device 200 differs from the first example above in that the training data generation unit 2004B, the machine learning unit 2005B, and the learned model LM2 are omitted.

[0227] The operation support unit 302 is a functional unit used to support users who operate the excavator 100 to perform operations or monitor the operation of the excavator 100.

[0228] The job support unit 302 includes a learned model storage unit 302A, a job object shape acquisition unit 302B, and a display processing unit 302E. That is, the job support unit 302 differs from the first example above in that the target trajectory generation unit 302C and the motion control unit 302D are omitted, and the display processing unit 302E is added.

[0229] The display processing unit 302E displays a screen related to work support provided to the user operating the excavator 100 to perform work or to monitor the excavator 100's work on the display device 50A. Furthermore, the display processing unit 302E can also transmit data corresponding to the same screen to the remote operation support device 400 or the remote monitoring support device via the communication device 60, and display it on the remote operation support device 400 or the remote monitoring support device.

[0230] For example, the display processing unit 302E displays an image representing the shape of the work object (ground) of the excavator 100 on the display device 50A based on the output of the work object shape acquisition unit 302B (data on the shape of the work object after the excavator 100 performs a predetermined action). Furthermore, the display processing unit 302E can also send the image data representing the shape of the work object of the excavator 100 to the remote operation support device 400 or the remote monitoring support device via the communication device 60, and display it on the remote operation support device 400 or the remote monitoring device. Thus, the user can grasp the shape of the work object of the excavator 100 while operating the excavator 100 or monitoring its operation by viewing the image representing the shape of the work object of the excavator 100 displayed on the display device 50A, etc. Furthermore, the display device 50A, etc., can update the image representing the shape of the work object of the excavator 100 in response to the execution of a predetermined action of the excavator 100. Therefore, users can grasp the shape of the object being worked on by the excavator 100 in real time and operate or monitor the excavator 100's work more accurately.

[0231] Thus, in this example, the controller 30 can more accurately grasp the shape of the sand and soil of the excavator 100's work object in the observation area. As a result, the user can more accurately grasp the shape of the sand and soil of the work object by displaying an image representing that shape. Therefore, the user can more accurately operate the excavator 100 or more accurately monitor the operation of the excavator 100 while grasping the shape of the sand and soil of the work object.

[0232] Furthermore, when the excavator 100 is operated remotely, some or all of the functions of the learned model storage unit 302A, the work object shape acquisition unit 302B, and the display processing unit 302E can be installed in the remote operation support device 400. Moreover, the functions of the work object shape acquisition unit 302B can be transferred to the information processing device 200. This reduces the processing load on the excavator 100.

[0233] <Example 3> Figure 8 This is the third example of a functional block diagram illustrating the functional structure of the SYS (System for Operation Support). Figure 9 This is a diagram illustrating a method for estimating the shape of a work object. Specifically, Figure 9 This diagram illustrates a method for estimating the shape of the work object after the excavator repeats a predetermined action 100 times (n times) from the initial shape of the work object. Figures 10-12 The figures show one example, another example, and yet another example of the measured and estimated data of the reaction force from the work object to the work part (bucket 6) during the digging action of the excavator 100. Figure 13 , Figure 14The figures show one and another example of the relationship between the accuracy of the estimation result of the shape of the object to be worked by the excavator 100 and the predetermined action of the excavator 100.

[0234] In addition, Figure 13 , Figure 14 In this context, the accuracy of the shape estimation result of the excavator 100's work object is defined by a value between the lowest "0" and the highest "1". Furthermore, in... Figure 13 In this context, the speed multiplier refers to a multiplier of the predetermined speed of the excavator 100, based on its normal operating state, i.e., when its operation is unrestricted. Furthermore, in... Figure 14 In this context, the digging depth ratio refers to the ratio of the digging depth of the excavator 100 to the normal state, i.e., when the operation of the excavator 100 is not restricted. Regarding the speed ratio and the digging depth ratio, the state where the operation of the excavator 100 is not restricted is set to "1", and when the operation of the excavator 100 is restricted, it is represented by a positive value less than "1".

[0235] The following figures will use the same reference numerals for structures that are the same as or correspond to those in Examples 1 and 2 above, and will focus on explaining the parts that are different from those in Examples 1 and 2 above.

[0236] Similar to Examples 1 and 2 above, the excavator 100 includes a support device 150. In this example, similar to Example 1 above, the support device 150 provides support related to the operation of the excavator 100, which operates through the autonomous operation function.

[0237] like Figure 8 As shown, similar to the first and second examples above, the support device 150 includes a controller 30, a hydraulic control valve 31, and sensors S1 to S9. In this example, unlike the first and second examples above, the excavator 100 is not equipped with sensor 40, and the support device 150 does not include sensor 40.

[0238] Similar to Examples 1 and 2 above, the controller 30 includes an action log providing unit 301 and an operation support unit 302 as functional units.

[0239] Similar to Examples 1 and 2 above, the action log providing unit 301 includes an action log recording unit 301A, an action log storage unit 301B, and an action log sending unit 301C.

[0240] Similar to Examples 1 and 2 above, the information processing device 200 includes a log acquisition unit 2001, a simulator unit 2002, a log storage unit 2003, a training data generation unit 2004, a machine learning unit 2005, a learned model storage unit 2006, and a distribution unit 2007 as functional units.

[0241] Regarding the information processing device 200, in this example, the difference from the first example is the addition of training data generation units 2004C and 2004D, machine learning units 2005C and 2005D, and the storage of learned models LM3 and LM4 in the learned model storage unit 2006. Furthermore, in this example, the difference from the first example is that the predetermined action of the excavator 100 is limited to the action of contacting the sand and soil of the work object with the work area. The predetermined action of the excavator 100 in this example includes, for example, a digging action. Moreover, the predetermined action of the excavator 100 in this example may also include a sweeping action, a compaction action, or a sweeping action.

[0242] The training data generation unit 2004 includes training data generation units 2004A to 2004D.

[0243] The training data generation unit 2004C generates a training dataset for generating the fully trained model LM3. The fully trained model LM3 infers the reaction force from the working object acting on the working part based on predetermined input data.

[0244] The input data corresponding to the learned model LM3 includes data representing the shape of the work object before the excavator 100 performs the predetermined action, and data representing the trajectory of the work part when the excavator 100 performs the predetermined action. Furthermore, the input data corresponding to the learned model LM3 may also include data representing the characteristics of the sand and soil of the work object of the excavator 100.

[0245] The training data is a combination of input data for the types specified in the learned model LM3 and data representing the correct inference results corresponding to that input data (correct data). Specifically, the input data included in the training data includes data representing the shape of the work object before the excavator 100 performs the predetermined action and data representing the trajectory of the working part when the excavator 100 performs the predetermined action. Furthermore, the input data included in the training data may also include data representing the characteristics of the sand and soil of the work object when the excavator 100 performs the predetermined action. Furthermore, the correct data included in the training data includes time-series data representing the reaction force exerted by the work object on the working part when the excavator 100 performs the predetermined action. Moreover, when multiple types of predetermined actions are specified, a learned model LM3 can be generated for each type of predetermined action. In this case, the training data generation unit 2004C generates a training dataset for each type of predetermined action.

[0246] The training data generation unit 2004D generates a training dataset for generating the learned model LM4. The learned model LM4 infers the accuracy of the shape estimation results of the excavator 100's work object. The shape estimation results of the excavator 100's work object include the shape estimation results of the excavator 100's work object identified by the work object shape acquisition unit 302B. Furthermore, the shape estimation results of the excavator 100's work object may include the initial shape of the work object identified by the work object shape acquisition unit 302B. This is because, in this example, the excavator 100 is not equipped with sensor 40, and the initial shape of the work object cannot be observed by the excavator 100. The accuracy of the shape estimation results of the excavator 100's work object refers to the degree of certainty of the shape estimation results of the excavator 100's work object. The accuracy of the shape estimation result of the excavator 100's work object is defined as follows: the smaller the difference between the estimated shape of the excavator 100's work object and the actual shape of the work object, the higher the accuracy; the larger the difference, the lower the accuracy. Furthermore, a relatively high accuracy in the shape estimation result of the excavator 100's work object means a relatively low uncertainty in the estimation result, and a relatively low accuracy in the estimation result means a relatively high uncertainty in the estimation result. Therefore, the accuracy of the shape estimation result of the excavator 100's work object also represents the degree of uncertainty in the shape estimation result of the excavator 100's work object. Moreover, for example, the accuracy of the shape estimation result of the excavator 100's work object is defined for each small region obtained by dividing the observation area of ​​the work object's shape into multiple regions. Therefore, the accuracy of the shape estimation result of the excavator 100's work object is, for example, represented as a vector defined by the accuracy of the shape estimation result of each small region of the excavator 100's work object.

[0247] For example, after learning the model LM4, the estimated shape of the work object before the excavator 100 performs a predetermined action is corrected using data obtained when the excavator 100 subsequently performs the predetermined action. This leads to the inference of the accuracy of the corrected shape of the work object's shape by the excavator 100. For example, as described later, the estimated shape of the work object before the excavator 100 performs the predetermined action is corrected based on the error between the measured result and the estimated result of the reaction force exerted by the work object on the work area when the excavator 100 performs the predetermined action. The shape of the work object before the excavator 100 performs the predetermined action refers, for example, the shape of the work object before the excavator 100 performs one predetermined action. Furthermore, the shape of the work object before the excavator 100 performs the predetermined action can be the shape of the excavator 100's work object before the start of a repeatedly performed predetermined action (i.e., the initial shape).

[0248] That is, after learning, the LM4 model can infer the accuracy of the estimated shape of the work object before the corrected excavator 100 performs the predetermined action, based on input data including data representing the estimated shape of the work object before the corrected excavator 100 performs the predetermined action.

[0249] The input data for the learned model LM4 includes data representing the estimated shape of the work object before the corrected excavator 100 performs the predetermined action, and data obtained when the excavator 100 performs the predetermined action. The data obtained when the excavator 100 performs the action includes, for example, data representing the trajectory of the work area when the excavator 100 performs the predetermined action. Furthermore, the data obtained when the excavator 100 performs the predetermined action may include data representing the reaction force exerted by the work object on the work area when the excavator 100 performs the predetermined action. Similarly to the learned model LM1, the input data for the learned model LM4 may include data representing the characteristics of the sand and soil of the work object of the excavator 100.

[0250] Training data is a combination of input data for the learned LM4 model and data representing the correct inference results corresponding to that input data (correct data). The training data includes correct data representing the accuracy of the shape of the work object before the excavator 100 performs the predetermined action, corresponding to the input data.

[0251] The Machine Learning Division 2005 includes Machine Learning Division 2005A to 2005D.

[0252] The Machine Learning Unit 2005C generates a fully learned model LM3 by performing machine learning on the basic learning model M3 based on the training dataset output from the training data generation unit 2004C. Furthermore, the Machine Learning Unit 2005C can also perform additional learning on the fully learned model LM3 to correct the error between the output (inference result) of the fully learned model LM3 and the actual reaction force corresponding to the shape of the work object.

[0253] The Machine Learning Division 2005D performs machine learning on the basic learning model M4 based on the training dataset output from the Training Data Generation Division 2004D to generate the fully learned model LM4. Furthermore, the Machine Learning Division 2005D can also perform supplementary learning on the fully learned model LM4 to correct the error between the output (inference result) of the fully learned model LM4 and the actual accuracy.

[0254] The learned models LM1 to LM4, which were output to the machine learning unit 2005, are stored in the learned model storage unit 2006. Furthermore, when the learned model LM1 is relearned or supplemented by the machine learning unit 2005A, the learned model LM1 in the learned model storage unit 2006 is updated. The same applies when the learned models LM2 to LM4 are relearned or supplemented by the machine learning units 2005B to 2005D, respectively. Moreover, the learned models LM1 to LM4 before the update can be saved in the learned model storage unit 2006 or other storage units in a way that allows for reuse. Therefore, for example, if there are problems with the updated learned models LM1 to LM4, the learned models LM1 to LM4 before the update can be recovered and reused.

[0255] The distribution department 2007 distributed the learned model data (LM1-LM4) to the excavator 100.

[0256] The distribution method for models LM3 and LM4 after learning can be the same as the distribution method for models LM1 and LM2 after learning.

[0257] Similar to the first example above, the operation support unit 302 is a functional unit that provides operation support to the excavator 100 that operates through the autonomous operation function.

[0258] The operation support unit 302 includes a learning completed model storage unit 302A, an operation object shape acquisition unit 302B, a target trajectory generation unit 302C, a motion control unit 302D, a reaction force estimation unit 302F, and an accuracy estimation unit 302G.

[0259] The learned models LM1 to LM4, which are distributed from the information processing device 200 and received through the communication device 60, are stored in the learned model storage unit 302A.

[0260] The reaction force estimation unit 302F estimates the reaction force acting on the working part when the excavator 100 performs the predetermined action based on the state of the work object before the excavator 100 performs the predetermined action and the trajectory of the working part when the excavator 100 performs the predetermined action.

[0261] For example, the reaction force estimation unit 302F estimates the reaction force acting on the working part when the excavator 100 performs the predetermined action, based on the shape of the work object before the excavator 100 performs the predetermined action and the track of the working part when the excavator 100 performs the predetermined action, using the learned model LM3.

[0262] The object shape acquisition unit 302B estimates the shape of the object to be worked by the excavator 100 based on the difference between the estimated result of the reaction force acting on the working part during the predetermined action of the excavator 100, estimated by the reaction force estimation unit 302F, and the actual measured result of the reaction force. Specifically, the object shape acquisition unit 302B corrects the estimated shape of the object to be worked before the excavator 100 performs the predetermined action based on the difference (estimation error) between the estimated result of the reaction force acting on the working part during the predetermined action of the excavator 100 and the actual measured result of the reaction force. The measured data of the reaction force of the excavator 100 are calculated, for example, based on the measurement data from sensors S7 to S9. Then, the object shape acquisition unit 302B estimates the shape of the object to be worked after the excavator 100 performs the predetermined action based on the corrected estimated shape of the object to be worked before the excavator 100 performs the predetermined action and the trajectory of the working part when the excavator 100 performs the predetermined action.

[0263] For example, such as Figure 9 As shown, the object shape acquisition unit 302B uses data representing the initial shape of the pre-acquired object as a starting point and employs a learned model LM1 to estimate the shape of the object after each predetermined action performed by the excavator 100. Hereinafter, to distinguish it from the actual initial shape, the initial shape corresponding to the data representing the initial shape of the object of the excavator 100 is sometimes referred to as the "temporary initial shape" for convenience. Furthermore, the reaction force estimation unit 302F uses a learned model LM3 to estimate the reaction force on the working part of the excavator 100 based on the estimation result (estimated sand shape) of the object's shape before each predetermined action performed by the excavator 100 and the data representing the trajectory of each working part. In this case, the data representing the estimation result of the object's shape before the excavator 100 performs the predetermined action is the data representing the initial shape of the object in the first predetermined action of the excavator 100. Furthermore, the trajectory of each working part can be data representing the target trajectory generated by the target trajectory generation unit 302C, or it can be measurement data based on the trajectory output of sensors S1 to S3.

[0264] Furthermore, the object shape acquisition unit 302B calculates the estimation error of the reaction force of the excavator 100 to the work site each time it performs a predetermined action based on the data of the estimated reaction force (estimated reaction force) estimated by the reaction force estimation unit 302F and the data of the actual reaction force measurement results.

[0265] Then, the object shape acquisition unit 302B corrects the data representing the initial shape of the object to be worked by the excavator 100 based on the estimation error of the reaction force to the working area each time the excavator 100 performs a predetermined action. Specifically, the object shape acquisition unit 302B applies a known mathematical programming algorithm to correct the temporary initial shape of the object to minimize the estimation error of the reaction force.

[0266] For example, the initial temporary shape of the sand or soil being worked on may have low accuracy, sometimes being set at a height lower than the actual initial shape. As a result, such as... Figure 10 As shown, during the predetermined action of the excavator 100, the working part may not make contact with the sand and soil of the work object and may spin idle, or, as... Figure 11 As shown, the working part of the excavator 100 may only come into contact with the sand and soil of the work object within a portion of the entire movement range that is to be contacted. Furthermore, the measured reaction force is smaller than the estimated reaction force throughout the entire predetermined movement of the excavator 100, and the estimation error between the estimated reaction force and the measured reaction force becomes a positive and relatively large value. In this case, the work object shape acquisition unit 302B corrects for at least the small area traversed by the working part during the predetermined movement of the excavator 100 within the observation area of ​​the work object, thereby increasing the ground elevation of the work object.

[0267] Furthermore, conversely, the accuracy of the provisional initial shape of the sand and soil being surveyed is low, sometimes set at a height higher than the actual initial shape. As a result, such as... Figure 12 As shown, during the predetermined action of the excavator 100, the working part may move along a track that passes through a position relatively deep above the ground of the work object. Furthermore, the measured reaction force becomes larger than the estimated reaction force from the initial stage of the excavator 100's predetermined action. In this example, when the measured reaction force reaches the upper limit of permissible value, the excavator 100 stops its predetermined action and performs an avoidance maneuver, moving along the working part in a direction away from the work object (refer to the dotted line portion in the figure). In this case, the work object shape acquisition unit 302B corrects for at least the small area (excluding the small area passed through by the avoidance maneuver) within the observation area of ​​the work object during the predetermined action of the excavator 100, so that the ground elevation of the work object is lowered.

[0268] The object shape acquisition unit 302B starts with data representing the initial shape of the corrected object and, based on data representing the trajectory of the working part at each predetermined action of the excavator 100, sequentially estimates the shape of the object after each predetermined action is performed by the excavator 100. Thus, the object shape acquisition unit 302B can estimate the shape of the object after the excavator 100 performs the nth predetermined action and acquire data representing the latest shape of the object.

[0269] The object shape acquisition unit 302B corrects the data representing the initial shape of the object each time the excavator 100 completes a predetermined action. Based on the corrected shape data, it estimates the latest shape of the object and acquires data representing that shape. Therefore, even if there is a deviation between the temporary initial shape and the actual initial shape at the start of the operation, the temporary initial shape can be used as a starting point, and the temporary initial shape can be corrected sequentially each time the excavator 100 performs a predetermined action. Thus, the object shape acquisition unit 302B sequentially reduces the error between the data representing the initial shape of the excavator 100's object and the data representing the actual initial shape, resulting in the ability to estimate the latest shape of the excavator 100's object with relatively high accuracy. Therefore, for example, even if measurement data representing the initial shape cannot be obtained externally, or if the measurement data representing the initial shape obtained externally is relatively inaccurate for some reason, the excavator 100 can utilize data representing the shape of the object with relatively high accuracy.

[0270] The accuracy estimation unit 302G estimates the accuracy of the shape estimation result of the excavator 100's work object. The shape estimation result of the excavator 100's work object can be the latest shape estimation result estimated by the work object shape acquisition unit 302B, or it can be the shape estimation result of the work object before the excavator 100 performs the predetermined action. Furthermore, the shape estimation result of the excavator 100's work object can also be a temporary initial shape of the excavator 100's work object. For example, the accuracy estimation unit 302G estimates the accuracy of the corrected shape of the work object before the excavator 100 performs the predetermined action based on data indicating the track of the working part when the excavator 100 performs the predetermined action or data indicating the reaction force acting on the working part. This is because, as described above, the shape of the work object before the excavator 100 performs the predetermined action is corrected based on data indicating the track of the working part when the excavator 100 performs the predetermined action or data indicating the reaction force acting on the working part. At this time, the accuracy of the estimated shape of the work object before the excavator 100 performs the predetermined action is, for example, the accuracy of the temporary initial shape of the work object after correction. Specifically, the accuracy estimation unit 302G uses the learned model LM4 to estimate the accuracy of the estimated shape of the work object before the excavator 100 performs the predetermined action after correction. Furthermore, the accuracy estimation unit 302G can also estimate the accuracy of the estimated shape of the work object before the excavator 100 performs the predetermined action based on rules. For example, the accuracy estimation unit 302G determines for each small area of ​​the observation area of ​​the work object whether the working part is in contact with the work object (ground) when the excavator 100 performs the predetermined action. Then, for each small area of ​​the object, the accuracy estimation unit 302G increases the accuracy relatively significantly from the current value if the working part is in contact with the work object, and increases the accuracy relatively slightly or maintains the current value if the working part is not in contact. This is because the shape of the work object before the excavator 100 performs the predetermined action is highly likely to be corrected with respect to the small area that the work part will contact when the excavator 100 performs the predetermined action.

[0271] The motion control unit 302D causes the excavator 100 to perform a predetermined action, moving the working part of the excavator 100 along the target track generated by the target track generation unit 302C. At this time, the motion control unit 302D can limit the action of the excavator 100 based on the accuracy of the estimation result of the shape of the object being worked on by the excavator 100, as estimated by the accuracy estimation unit 302G. This is because if the accuracy of the estimation result of the shape of the object being worked on by the excavator 100 is relatively low, the reaction force acting on the working part becomes problematic due to the mismatch between the estimated shape of the object being worked on and the target track, potentially causing a large impact on the excavator 100. Specifically, the motion control unit 302D can calculate the average accuracy (average accuracy) of a small area located on the target track when viewed from above. Then, the motion control unit 302D can increase the degree of restriction on the action of the excavator 100 when the average accuracy is relatively low, and decrease the degree of restriction on the action of the excavator 100 when the average accuracy is relatively high.

[0272] For example, such as Figure 13 As shown, the motion control unit 302D changes the speed multiplier corresponding to the predetermined speed of the excavator 100 based on the average accuracy. In this example, the speed multiplier is set to 1 when the average accuracy exceeds the threshold TH1 (<1), and is set to decrease linearly with decreasing average accuracy when the average accuracy is below the threshold TH1. Therefore, when the average accuracy is below the threshold TH1, the motion control unit 302D can limit the operating speed of the excavator 100, causing the excavator 100 to perform the predetermined action at a slower speed than when there is no motion limitation. Therefore, even if the estimated shape of the workpiece of the excavator 100 does not match the target track, and the working part unintentionally comes into contact with the workpiece, the impact on the excavator 100 can be suppressed to a relatively small extent.

[0273] And, for example, such as Figure 14As shown, the motion control unit 302D changes the digging depth ratio corresponding to the digging action of the excavator 100 based on the average accuracy. In this example, the digging depth ratio is set to 1 when the average accuracy exceeds the threshold TH2 (<1), and is set to decrease linearly relative to the decrease in average accuracy when the average accuracy is below the threshold TH2. Therefore, when the average accuracy is below the threshold TH2, the motion control unit 302D, by limiting the digging depth during the digging action of the excavator 100, can further limit the range of movement of the working part towards the target object (ground side) compared to when there is no motion restriction. Therefore, even if the accuracy of the estimated shape of the target object is low and the actual ground height is higher than expected, it can suppress the relatively large impact on the excavator 100 caused by the working part penetrating very deep into the ground. For example, the motion control unit 302D limits the digging depth by correcting the target track generated by the target track generation unit 302C. Furthermore, the target track generation unit 302C can generate a target track with a digging depth corresponding to the average accuracy, and the motion control unit 302D can cause the excavator 100 to perform a predetermined action based on the target track. As a result, the motion control unit 302D limits the digging depth.

[0274] In addition, the motion control unit 302D can also limit the range of motion of the working part of the excavator 100 in the vertical direction during the predetermined action of the excavator 100, other than the digging action, based on the average accuracy.

[0275] Thus, in this example, the controller 30 acquires data representing the state of the excavator 100 and related to the shape of the work object in accordance with the predetermined action of the excavator 100. Then, the controller 30 estimates the shape of the sand and soil on the work object (ground) of the excavator 100 based on the acquired data. Specifically, the controller 30 acquires data representing the trajectory of the excavator 100 when performing the predetermined action, and estimates the reaction force acting on the work area when the excavator 100 performs the predetermined action based on this data. Furthermore, the controller 30 acquires measurement data of the reaction force acting on the work area when the excavator 100 performs the predetermined action, and corrects the shape of the work object before the excavator 100 performs the predetermined action based on the difference (estimation error) between the measured data and the estimated data. Then, the controller 30 estimates the shape of the work object after the excavator 100 performs the predetermined action based on the corrected data representing the shape of the work object before the excavator 100 performs the predetermined action and the data representing the trajectory of the excavator 100 when performing the predetermined action. Therefore, even when the accuracy of the shape of the work object before the excavator 100 performs the predetermined action is low, the shape of the work object can be corrected based on the predetermined action of the excavator 100. Consequently, the controller 30 can more accurately grasp the shape of the sand and soil of the work object in the observation area, and as a result, can more accurately generate the target trajectory of the excavator 100's working part. Therefore, the controller 30 enables the excavator 100 to perform autonomous operation more accurately.

[0276] Furthermore, some or all of the functions of the reaction force estimation unit 302F and the accuracy estimation unit 302G can be transferred to the information processing device 200. As a result, the processing load of the excavator 100 can be reduced.

[0277] <Another example> Examples 1 to 3 of the functional structure of the aforementioned operational support system SYS can be appropriately modified or altered.

[0278] For example, in the second example of the functional structure of the above-described operation support system SYS, the excavator 100 can omit the sensor 40, and the object shape acquisition unit 302B can estimate the shape of the object to be worked by the excavator 100 using the same method as in the third example, and acquire data representing that shape. In this case, the same functional unit as the reaction force estimation unit 302F in the third example is added to the operation support unit 302. Furthermore, in this case, the functions of the accuracy estimation unit 302G in the third example and a portion of the functions of the motion control unit 302D can also be added to the operation support unit 302, namely, the function of limiting the predetermined actions of the excavator 100 based on the accuracy of the estimation result of the shape of the object to be worked by the excavator 100.

[0279] Furthermore, in the third example of the functional structure of the above-mentioned operation support system SYS, the functions of the accuracy estimation unit 302G and part of the functions of the motion control unit 302D (the function of limiting the predetermined actions of the excavator 100 based on the accuracy of the estimation result of the shape of the work object of the excavator 100) can be omitted.

[0280] [Specific examples of processing related to the use of data representing the shape of the work object] Next, refer to Figures 15-18 A specific example of processing related to the use of data representing the shape of the work object of the excavator 100 acquired by the work object shape acquisition unit 302B will be explained.

[0281] <Example 1> Figure 15 This is a diagram illustrating the first example of processing related to the use of data representing the shape of a work object. Specifically, Figure 15 This is a diagram illustrating an example of how the controller 30 enables the excavator 100 to operate autonomously to perform excavation work on the target area based on data representing the shape of the target object.

[0282] Furthermore, the excavation operation is performed through a series of actions including repeated digging, boom lifting and slewing, dumping, and boom lowering and slewing. The digging action is the action of the attachment AT used to excavate sand from the work area using the bucket 6. For example, the digging action is achieved as a composite action of at least two driven components of the boom 4, stick 5, and bucket 6. The boom lifting and slewing action is the action of the attachment AT and the upper slewing body 3 used to scoop up the sand excavated in the digging action with the bucket 6 and transport it to a dumping position away from the work area. For example, the boom lifting and slewing action is achieved as a composite action of the lifting action of the boom 4 and the slewing action of the upper slewing body 3. The dumping action is the action of the attachment AT used to dump the sand from the bucket 6 onto the ground at the dumping position. The dumping action is, for example, achieved as a composite action of the opening action of the stick 5 and the opening action of the bucket 6. The boom lowering and slewing action is the action of the attachment AT and the upper slewing body 3 used to return the bucket 6 to the work area. The boom lowering and slewing motion is achieved, for example, as a combination of the lowering motion of the boom 4 and the slewing motion of the upper slewing body 3.

[0283] This flowchart begins, for example, when an input indicating the start of excavation work via the autonomous operation of the excavator 100 is received. This input can be made by the operator of the cab 10, for example, via the input device 52, or from outside the excavator 100 (information processing device 200 or remote operation support device 400) via the communication device 60. The same applies to the input indicating the start of loading work via the autonomous operation of the excavator 100.

[0284] like Figure 15 As shown, in step S10, the controller 30 sets the target area (operation target area) for the excavation operation. The target area for the excavation operation is defined, for example, by the overhead view of the excavation operation and the depth of the excavation. The overhead view of the excavation operation in the operation target area corresponds to the observation target area of ​​the operation target shape acquisition unit 302B. The operation target area can be set by input from the operator of the cab 10 via the input device 52, or by input from outside the excavator 100 (e.g., the information processing device 200 or the remote operation support device 400) via the communication device 60.

[0285] If step S10 is completed, the controller 30 proceeds to step S11.

[0286] In step S11, the controller 30 sets the initial shape of the work object. For example, when the sensor 40 is mounted on the excavator 100, the initial shape of the work object is set based on measurement data acquired by the sensor 40 before the start of work. Furthermore, when the sensor 40 is not mounted on the excavator 100, the initial shape of the work object can be set to a pre-defined temporary shape. The temporary shape is, for example, a planar shape extending the ground plane of the lower walking body 1 of the excavator 100 to the work object area. The initial shape of the work object can also be set manually based on input from an operator or similar source. Additionally, the initial shape of the work object can be set based on measurement data of the shape of the work object around the excavator 100 input from an external source (e.g., an information processing device 200 or a remote operation support device 400) via the communication device 60.

[0287] If step S11 is completed, the controller 30 proceeds to step S12.

[0288] In step S12, the target track generation unit 302C of the controller 30 generates the target track of the working part of the excavator 100 based on the data representing the estimated shape of the current work object.

[0289] The data representing the estimated shape of the current work object, when transitioning from step S11 to step S12, represents the initial shape of the work object set in step S12. Furthermore, the data representing the estimated shape of the current work object, when transitioning from step S16 to step S12, represents the latest shape of the work object acquired by the work object shape acquisition unit 302B.

[0290] If step S12 is completed, the controller 30 proceeds to step S13.

[0291] In step S13, the motion control unit 302D of the controller 30 sets the motion conditions for the excavator 100's digging action. For example, the motion control unit 302D can set the speed conditions for the digging action based on the accuracy of the estimation result of the shape of the work object estimated by the accuracy estimation unit 302G. Furthermore, the motion control unit 302D can also set the digging depth conditions for the digging action based on the accuracy of the estimation result of the shape of the work object estimated by the accuracy estimation unit 302G.

[0292] If step S13 is completed, the controller 30 proceeds to step S14.

[0293] In step S14, the motion control unit 302D executes the digging action of the excavator 100 according to the target track generated in step S12 and the motion conditions set in step S13.

[0294] If step S14 is completed, the controller 30 proceeds to step S15.

[0295] In step S15, the controller 30 processes data (sand shape data) representing the shape of the work object within the observation area defined according to the work object area set in step S10. Details of this processing will be described later.

[0296] If step S15 is completed, the controller 30 proceeds to step S16.

[0297] In step S16, the controller 30 determines whether the condition indicating the end of the operation (operation end condition) is met. For example, the controller 30 compares the depth setting of the operation target area with the height position of the sand in the operation target area in the latest sand shape data obtained in step S15. If the excavation has reached the set depth of the operation target area, the operation end condition is determined to be met. If the operation end condition is not met, the controller 30 returns to step S12 and repeats the processing from step S12 to step S16. If the operation end condition is met, the current flowchart processing ends.

[0298] Thus, in this example, the controller 30 enables the excavator 100 to autonomously perform excavation operations targeting the work area.

[0299] <Example 2> Figure 16 This is a second example of a process related to the use of data representing the shape of a work object. Specifically, Figure 16 This is an example of a process used by the controller 30 to enable the excavator 100 to operate autonomously to perform a loading operation of sand onto a truck based on data representing the shape of the work object.

[0300] Furthermore, the loading operation onto the truck is performed through a series of actions including repeated digging, boom lifting and slewing, dumping, and boom lowering and slewing. The digging action is the action of attachment AT used to scoop sand from a predetermined location on the ground into the bucket 6. For example, the digging action is achieved as a combined action of at least two driven components of the boom 4, stick 5, and bucket 6. The boom lifting and slewing action is the action of attachment AT and upper slewing body 3 used to transport the sand scooped into the bucket 6 onto the truck bed. For example, the boom lifting and slewing action is achieved as a combined action of the lifting action of the boom 4 and the slewing action of the upper slewing body 3. The dumping action is the action of attachment AT used to dump sand from the bucket 6 into the truck bed. The dumping action is achieved, for example, as a combined action of the opening action of the stick 5 and the opening action of the bucket 6. The boom lowering and slewing action is the action of attachment AT and upper slewing body 3 used to return the bucket 6 to the predetermined position for scooping sand. The boom lowering and slewing motion is achieved, for example, as a combined motion of lowering the boom 4 and slewing the upper slewing body 3. Furthermore, in this example, the description assumes that the location and shape of the sand pile (sand heap) generated as a result of the excavation operation are known. This is because the location where the excavator 100 discharges soil during the excavation operation is predetermined based on the excavator 100, and the approximate shape of the sand heap generated, taking into account the size or depth of the work area, can also be roughly envisioned.

[0301] This flowchart, for example, begins when an input indicating that loading operations have commenced via the autonomous operation of excavator 100 is received.

[0302] like Figure 16 As shown, in step S20, the controller 30 detects the relative position of the truck carrying the sand relative to the excavator 100. For example, if the absolute positions of the excavator 100 and the truck are known, the controller 30 detects the relative position of the truck relative to the excavator 100 based on their absolute positions. In this case, the controller 30 detects the relative position of the truck relative to the excavator 100, for example, based on the output of the GNSS sensor mounted on the excavator 100 and the truck's position information received from the truck via the communication device 60. Furthermore, the controller 30 can also receive information about the relative position of the truck relative to the excavator 100 from an external device (e.g., an information processing device 200, etc.) via the communication device 60, obtained from the output of a camera or distance sensor installed at the construction site. Furthermore, if the excavator 100 is equipped with sensor S6, the controller 30 can identify the truck based on the output of sensor S6, thereby detecting the relative position of the truck relative to the excavator 100.

[0303] If step S20 is completed, the controller 30 proceeds to step S21.

[0304] In step S21, the controller 30 sets the initial shape of the work object. In this example, the controller 30 estimates the position of the truck bed based on the position of the truck relative to the excavator 100 detected in step S20, and sets the bottom surface of the truck bed as the work object, thus setting its initial shape.

[0305] Furthermore, if the truck's parking orientation relative to the excavator 100 has not yet been determined, the truck's relative orientation relative to the excavator 100 can be detected in step S20. For example, if the orientations of both the excavator 100 and the truck are known, the controller 30 detects the truck's relative orientation relative to the excavator 100 based on their respective orientations. In this case, the controller 30 detects the truck's orientation relative to the excavator 100, for example, based on the output of the orientation sensor and the truck's orientation information received from the truck via the communication device 60. The controller 30 can also receive information about the truck's relative orientation relative to the excavator 100 from an external device (e.g., an information processing device 200, etc.) via the communication device 60, obtained from the output of a camera or distance sensor installed at the construction site. Furthermore, if the excavator 100 is equipped with a sensor S6, the controller 30 can identify the truck based on the output of the sensor S6, thereby detecting the truck's relative orientation relative to the excavator 100.

[0306] If step S21 is completed, the controller 30 proceeds to step S22.

[0307] In step S22, the target track generation unit 302C of the controller 30 generates the target track of the working part of the excavator 100 based on the data representing the estimated shape of the current work object.

[0308] The data representing the estimated shape of the current work object, when transitioning from step S21 to step S22, represents the initial shape of the work object set in step S22. Furthermore, the data representing the estimated shape of the current work object, when transitioning from step S25 to step S22, represents the latest shape of the work object acquired by the work object shape acquisition unit 302B.

[0309] If step S22 is completed, the controller 30 proceeds to step S23.

[0310] In step S23, the motion control unit 302D of the controller 30 causes the excavator 100 to perform a series of actions along the target track generated in step S22, consisting of a combination of digging action, boom lifting and slewing action, soil discharge action and boom lowering and slewing action.

[0311] If step S23 is completed, the controller 30 proceeds to step S24.

[0312] In step S24, the controller 30 processes data (sand shape data) representing the shape of the work object for the observation area corresponding to the truck cargo box detected in step S20. Details of this processing will be described later.

[0313] If step S24 is completed, the controller 30 proceeds to step S26.

[0314] In step S26, the controller 30 determines whether the condition indicating the end of the operation (operation end condition) is met. For example, based on data indicating the shape of the work object, if the height of the sand above the bottom of the cargo box is relatively large compared to a predetermined reference, the controller 30 determines that the loading operation of loading sand onto the truck is complete. If the operation end condition is met, the controller 30 ends the processing of this flowchart; if the operation end condition is not met, it returns to step S22 and repeats the processing up to steps S22 to S26.

[0315] Furthermore, in this example, in step S24, in addition to acquiring data representing the shape of the sand in the truck bed (sand shape data), data representing the shape of the dump pile (sand shape data) can also be acquired as data representing the shape of the work object. In this case, in step S22, the target track generation unit 302C of the controller 30 generates a series of target tracks based on the sand shape data of the dump pile and the sand shape data of the truck bed acquired in step S24.

[0316] Thus, in this example, controller 30 is able to perform the loading operation autonomously.

[0317] <Example 3> Figure 17 This is a diagram illustrating the third example of processing related to the use of data representing the shape of the work object. Specifically, Figure 17 This diagram illustrates an example of the processing by which the controller 30 displays an image representing the shape of the work object on a display device 50A, etc., based on data representing the shape of the work object, while the excavator 100 is performing an excavation operation.

[0318] This flowchart begins, for example, when input indicating that the operator should manually begin the excavation operation is received. This input can be made by the operator in the cab 10 via input device 52, or by a remote operator via remote operation support device 400. In the latter case, the input from remote operation support device 400 is received by controller 30 via communication device 60.

[0319] like Figure 17 As shown, the processing in steps S30 and S31 is similar to... Figure 15 The processes in steps S10 and S11 are the same, so the explanation is omitted.

[0320] If step S31 is completed, the controller 30 proceeds to step S32.

[0321] In step S32, the display processing unit 302E of the controller 30 displays an image (initial shape image) representing the initial shape of the sand of the work object on the display device 50A or the display device of the remote operation support device 400, based on the data representing the initial shape set in step S31.

[0322] Furthermore, the display processing unit 302E of the controller 30 sends a display command containing data representing an image showing the initial shape to the remote operation support device 400 via the communication device 60, thereby displaying the initial shape image on the display device of the remote operation support device 400. (The following will be discussed further.) Figure 18 Step S42 can also be the same.

[0323] If step S32 is completed, the controller 30 proceeds to step S33.

[0324] In step S33, the controller 30 monitors the operation of the excavator 100 based on the outputs of sensors S1 to S9.

[0325] If step S33 is completed, the controller 30 proceeds to step S34.

[0326] In step S34, the controller 30 determines whether the digging action has been executed and completed. If the digging action has been executed and completed, the controller 30 proceeds to step S35; otherwise, the controller 30 proceeds to step S37.

[0327] In step S35, the controller 30 processes data (sand shape data) representing the shape of the work object within the observation area defined according to the work object area set in step S30. Details of this processing will be described later.

[0328] If step S35 is completed, the controller 30 proceeds to step S36.

[0329] In step S36, the display processing unit 302E of the controller 30 updates the image (sand shape image) representing the shape of the sand of the work object displayed on the display device 50A or the display device of the remote operation support device 400 based on the sand shape data obtained in step S35.

[0330] Furthermore, the display processing unit 302E of the controller 30 sends an update command containing data representing the latest sand shape image to the remote operation support device 400 via the communication device 60, thereby updating the sand shape image on the display device of the remote operation support device 400. Hereinafter, for the purposes described later... Figure 18 Step S46 can also be the same.

[0331] If step S36 is completed, the controller 30 proceeds to step S37.

[0332] In step S37, the controller 30 determines whether the condition for ending the display of the sand shape image (display end condition) is met. For example, the display end condition is the receipt of a predetermined input from an operator or the like via the input device 52 or the input device of the remote operation support device 400. Furthermore, the display end condition can also be that the state of not performing digging operations continues for a certain period of time. If the display end condition is not met, the controller 30 returns to step S33 and repeats the processing up to steps S33 to S37. If the display end condition is met, the current flowchart processing ends.

[0333] Thus, in this example, the controller 30 can update the image representing the shape of the sand in the work area according to the progress of the excavation work in the work area, and display it on the display device 50A or the display device of the remote operation support device 400.

[0334] <Example 4> Figure 18 This is the fourth example of a process related to the use of data representing the shape of a work object. Specifically, Figure 18 This diagram illustrates an example of the processing by which the controller 30 displays an image representing the shape of the work object on a display device 50A, etc., based on data representing the shape of the work object, when the excavator 100 is performing a loading operation to load sand onto a truck.

[0335] This flowchart begins, for example, when an input indicating that the loading operation should be started manually by the operator is received. This input can be made by the operator in the cab 10, for example, via input device 52, or by a remote operator via remote operation support device 400. In the latter case, the input from remote operation support device 400 is received by controller 30 via communication device 60.

[0336] like Figure 18 As shown, the processing in steps S40 and S41 is similar to... Figure 16 The processes in steps S20 and S21 are the same, so the explanation is omitted.

[0337] If step S41 is completed, the controller 30 proceeds to step S42.

[0338] In step S42, the display processing unit 302E of the controller 30 displays an image (initial shape image) representing the initial shape of the sand of the work object on the display device 50A or the display device of the remote operation support device 400, based on the data representing the initial shape set in step S41.

[0339] If step S42 is completed, the controller 30 proceeds to step S43.

[0340] In step S43, the controller 30 monitors the operation of the excavator 100 based on the outputs of sensors S1 to S9.

[0341] If step S43 is completed, the controller 30 proceeds to step S44.

[0342] In step S44, the controller 30 determines whether the soil removal action has been executed and completed. If the soil removal action has been executed and completed, the controller 30 proceeds to step S45; otherwise, it proceeds to step S47.

[0343] In step S45, the controller 30 processes data (sand shape data) representing the shape of the work object for the observation area corresponding to the truck's cargo box detected in step S40. Details of this processing will be described later.

[0344] If step S45 is completed, the controller 30 proceeds to step S46.

[0345] In step S46, the display processing unit 302E of the controller 30 updates the image (sand shape image) representing the shape of the sand of the work object displayed on the display device 50A or the display device of the remote operation support device 400 based on the sand shape data obtained in step S45.

[0346] If step S46 is completed, the controller 30 proceeds to step S47.

[0347] In step S47, the controller 30 determines whether the condition for ending the display of the sand shape image (display end condition) is met. For example, the display end condition is receiving a predetermined input from an operator or the like via the input device 52 or the input device of the remote operation support device 400. Furthermore, the display end condition can also be that the state of not performing the soil removal action continues for a certain period of time. If the display end condition is not met, the controller 30 returns to step S43 and repeats the processing up to steps S43 to S47. If the display end condition is met, the current flowchart processing ends.

[0348] Thus, in this example, the controller 30 can update the image representing the shape of the sand in the truck bed according to the progress of the loading operation of loading sand into the truck bed, which is the object of the operation, and display it on the display device 50A or the display device of the remote operation support device 400.

[0349] [Specific examples of processing related to acquiring data representing the shape of the work object] refer to Figures 16-18 A specific example of the processing of the object shape acquisition unit 302B will be explained.

[0350] <Example 1> Figure 19 This is a flowchart illustrating, schematically, a first example of a process for obtaining data representing the shape of a work object. Figure 20 This is a diagram showing an example of the area being observed. Figure 21 This is a diagram showing an example of the affected area.

[0351] In this example, the first example of the functional structure of the aforementioned operation support system SYS is used. Figure 6 ) or the second case ( Figure 7 This explanation is based on the premise that the predetermined action of the excavator 100 is an excavation action.

[0352] Processing for obtaining the shape of the work object Figure 19 The flowchart, for example, is executed after the excavator 100 performs the digging action. Specifically, Figure 19 The flowchart is equivalent to the above Figure 15 Step S15 processing or Figure 17 The processing of step S35. Hereinafter, in this flowchart, it will be explained on the premise of obtaining data representing the shape (sand shape) of the work object after the excavator 100 performs the k-th (k: a positive integer) excavation action from the start of the excavation operation.

[0353] For example, such as Figure 20As shown, the observation area TA around the excavator 100 is divided into a predetermined number of grids N. The observation area TA is the region around the excavator 100 where the object shape acquisition unit 302B acquires data representing the shape of the sand. In this example, as data representing the shape of the sand, the height h of the sand in each grid i (i=1 to N) of the observation area TA is acquired. e k and the uncertainty s corresponding to its estimation accuracy e k The height h of the sand e k and uncertainty s e k It is represented by the following formulas (1) and (2).

[0354] [Formula 1] In step S102, the object shape acquisition unit 302B inputs data on the shape of the sand before the excavator 100 performs the digging action. The data on the shape of the sand before the excavator 100 performs the digging action is equivalent to the data on the shape of the sand after the (k-1)th digging action (the height h of the sand). e k-1 and uncertainty s e k-1 ).

[0355] Furthermore, in the case of the first excavation operation (k=1), the initial value of the sand shape data (sand height h) is input into the object shape acquisition unit 302B. e 0 and uncertainty s e 0 ).

[0356] Additionally, the initial value of the sand shape data can be obtained from the output of sensor 40 (shape sensor), from outside the excavator 100, or from a pre-defined assumption. A pre-defined assumption, for example, is zero (0) assuming the entire observation area TA (all grids i) is at the same height as the ground at the location of the excavator 100.

[0357] If the processing in step S102 is completed, the object shape acquisition unit 302B proceeds to step S104.

[0358] In step S104, the object shape acquisition unit 302B inputs the log of the current excavation action of the excavator 100 for each grid i of the observation object area TA. The log includes the height b of the cutting edge (tooth tip) of the bucket 6. k The angle of the cutting edge (tooth tip) of bucket 6 、 Excavation reaction force f k Sandy soil characteristics λ k and impact information κ k Furthermore, logs can also include other information. k .

[0359] The height b of the cutting edge of bucket 6 k It refers to the height through which the tip of the bucket 6 passes in each grid i of the observation area TA during the current digging action of the excavator 100, and is represented by the following formula (3).

[0360] [Equation 2] Regarding cell i corresponding to the position that bucket 6 did not pass through, the height b of the cutting edge of bucket 6 does not need to be entered. k Alternatively, an appropriate value can be entered. Regarding the angle of the cutting edge of bucket 6, which will be discussed later... and the excavation reaction force f k The same applies. The height b of the cutting edge of bucket 6. k For example, the data can be obtained from the trajectory data of the bucket 6 during the current digging action of the excavator 100. The trajectory data of the bucket 6 can be obtained, for example, from the timing data output by sensors S1 to S6 during the current digging action of the excavator 100.

[0361] The angle of the cutting edge of bucket 6 The angle between the tip of the bucket 6 and a predetermined reference plane (e.g., the horizontal plane) when the tip of the bucket 6 passes through each grid i in the observation area TA during the current excavation action of the excavator 100 is represented by the following formula (4).

[0362] [Formula 3] The angle of the cutting edge of bucket 6 For example, the data can be obtained based on the trajectory data of the bucket 6 and the timing data of the bucket 6's posture angle during the current digging action of the excavator 100. The timing data of the bucket 6's posture angle can be obtained, for example, based on the timing data output by sensor S3 during the current digging action of the excavator 100.

[0363] Excavation reaction force f k It refers to the reaction force from the ground acting on the bucket 6 in each grid i of the observation area TA during the current excavation action of the excavator 100, and is expressed by the following formula (5).

[0364] [Formula 4] Excavation reaction force f kFor example, it can be obtained based on the timing data output by sensors S7 to S9 during the current digging action of the excavator 100.

[0365] Furthermore, the digging reaction force of each cell i can be represented as a vector. In this case, the sand and soil properties λ of each cell i in the observed area TA are... i k It's not one-dimensional, but three-dimensional.

[0366] Sandy soil characteristics λ k It refers to the characteristics of the sand in each cell i of the observed object area TA, which is represented by the following formula (6).

[0367] [Formula 5] Sandy soil characteristics λ k For example, the angle of repose of each cell i in the observed region TA. Sand characteristics λ k For example, it is predetermined. Furthermore, the sandy soil characteristic λ k It can also be obtained based on the timing data of the output of sensor 40 or the output of sensors S7 to S9.

[0368] Furthermore, each grid cell i can have multiple types of sand and soil properties. In this case, the sand and soil properties λ of each grid cell i in the observed region TA are... i k It's not one-dimensional, but multi-dimensional.

[0369] Impact Information κ k This indicates whether the current digging action of excavator 100 in each cell i of the observed area TA has a direct impact on the shape of the sand. For example, the impact information κ k Whether the tip of the bucket 6 passes through each cell i in the observed area TA is represented by the following formula (7).

[0370] [Formula 6] For example, during this excavation operation, the cutting edge of bucket 6 passes through grid i (i=1, ..., N) of the observation area TA, and the height of the cutting edge of bucket 6 at this time is the height h of the sand. e，i k-1 In the following situations, information κ is affected i k Set to "+1" (κ) i k =+1). On the other hand, during this excavation operation, the tip of bucket 6 did not pass through grid i (i=1, ..., N) of the observed area TA, or although it did pass through, its height at that time was higher than the height h of the sand. e，ik-1 In this case, the information κ is affected i k Set to "-1" (κ) i k =-1).

[0371] Other information ω k For example, the weight w of the sand contained in the bucket 6 after the excavator 100 performs the digging action. k and volume v k , is represented by the following formula (8).

[0372] [Formula 7] If the processing in step S104 is completed, the object shape acquisition unit 302B proceeds to step S106.

[0373] In step S106, the object shape acquisition unit 302B performs uncertainty s of the sand shape. e，i k-1 Update.

[0374] For example, the object shape acquisition unit 302B is set to obtain the shape of the sand (height h of the sand) based on the current digging action of the excavator 100. e，i k The area affected by the operation (hereinafter referred to as the "affected area") is Ω. Then, the object shape acquisition unit 302B determines the uncertainty s of the sand shape for each grid i contained in the affected area Ω. e，i k Improvement, and updates are performed using the following formula (9).

[0375] [Formula 8] For example, such as Figure 21 As shown, the influence area Ω is set to be the range within which the excavation area EA expands only by a predetermined amount in all four directions. The excavation area EA is the area within the observation target area TA where the tip of the bucket 6 passes through during the current excavation action of the excavator 100, and the height of the tip of the bucket 6 at this time is the height h of the sand. e，i k-1 The region corresponding to the set of the following lattice i.

[0376] Uncertainty s e，i k The increase c i For example, the value is the same in every cell of the influence region Ω. Furthermore, the uncertainty s e，i k The increase c iIt can also be set such that, within the influence area Ω, the excavation area EA is the largest and decreases as one moves away from the excavation area EA.

[0377] If the processing in step S106 is completed, the object shape acquisition unit 302B proceeds to step S108.

[0378] In step S108, the object shape acquisition unit 302B determines the shape of the sand before the excavator 100 performs the current excavation operation based on the output of the sensor 40. Therefore, the object shape acquisition unit 302B can determine the shape of the sand after the excavator 100 performs the current excavation operation for the grid i in the observation area TA where the sensor 40 can measure the height of the sand.

[0379] For example, the object shape acquisition unit 302B uses a Kalman filter to correct the shape of the sand after the excavator 100 performed the last digging action based on the output of the sensor 40, thereby determining the shape of the sand after the excavator 100 performs the current digging action.

[0380] If a Kalman filter is applied, then for any cell i in the observed object area TA after the excavator 100 performs the digging action, with variance s i The observed height z of the sand i At that time, the height h of the sand e，i k and its uncertainty s e，i k It is represented by the following formulas (10) to (14).

[0381] [Formula 9] If equations (10) to (14) are summarized as function K, then the following equation represents the expression.

[0382] [Formula 10] Therefore, for grid i in the observed region TA, the variance s is measured by sensor 40. l，i Measure the height z of the sand. l，i At that time, the height h of the sand based on the output of sensor 40 l，i k and uncertainty s l，i k It is represented by the following formula (16).

[0383] [Equation 11] Therefore, the object shape acquisition unit 302B is able to acquire (determine) the height h of the sand based on the output of the sensor 40 after the excavator 100 performs the current digging action. l，i k and uncertainty s l，i k .

[0384] The height h of the sand in each cell i in the observation area TA, based on the output of sensor 40. l k It is represented by the following formula (17).

[0385] [Equation 12] Sensor 40 may sometimes be unable to measure the height of all cells i in the observation area TA due to obstruction. Therefore, the height h of the sand is obtained only for the cells i whose height can be measured by sensor 40 using equation (16). l，i k and uncertainty s l，i k For grid i, where the sand height cannot be measured by sensor 40, the sand height h is... l，i k and uncertainty s l，i k For example, maintaining the previous value (the height of the sand, h). l，i k-1 and uncertainty s l，i k-1 Furthermore, for grid i where the height of the sand cannot be measured by sensor 40, the height h of the sand is... l，i k and uncertainty s l，i k This can be specified as an undefined value ("unknown"). Below, we use measurement capability information m, representing whether sensor 40 can measure the shape of the sand in each cell i of the observed area TA. l k For example, whether the measurement can provide information m. l k It is represented by the following formulas (18) to (20).

[0386] [Equation 13] If the processing in step S108 is completed, the object shape acquisition unit 302B proceeds to step S110.

[0387] In step S110, the object shape acquisition unit 302B uses a function g, equivalent to the learned model LM1, to infer the sand shape (height z of the sand) of each cell i. p k and its uncertainty s p k ).

[0388] [Formula 14] The function g is constructed, for example, around a deep neural network (DNN). Furthermore, it is possible to set the input and output to an image format and use U-Net. In this case, the other information ω in equation (8) above... k It does not use an image format, so it can be expanded into an image format or directly input into the intermediate layer.

[0389] Furthermore, similar to step S108, the object shape acquisition unit 302B applies a Kalman filter to estimate the shape of the sand after the excavator 100 performs this excavation action based on the inference result of function g. For grid i in the observed object area, the height h of the sand is calculated based on the inference result of function g. p，i k and uncertainty s p，i k It is represented by the following formula (22).

[0390] [Formula 15] If the processing in step S110 is completed, the object shape acquisition unit 302B proceeds to step S112.

[0391] In step S112, the object shape acquisition unit 302B generates data (sand height h) representing the final shape of the sand after the excavator 100 performs the digging action, based on the outputs of steps S108 and S110. e k and uncertainty s e k That is, the object shape acquisition unit 302B integrates the sand shape data output in step S108 based on the output of sensor 40 and the sand shape data output in step S110 based on the reasoning result of function g.

[0392] For example, the object shape acquisition unit 302B will use the sand shape data (sand height h) determined in step S108 based on the output of sensor 40. l，i k and uncertainty sl，i k ) as the final data for the shape of the sand (the height h of the sand) e，i k and uncertainty s e，i k ).

[0393] However, as mentioned above, sometimes due to obstruction of the sensor 40, the shape of the sand cannot be determined based on the output of the sensor 40 during the processing in step S108, i.e., the measurement availability information m is lost. l k The value of cell i is "-1". Therefore, for cell i, the object shape acquisition unit 302B will obtain the sand shape data (sand height h) determined in step S110 based on the reasoning result of function g. p，i k and uncertainty s p，i k (This serves as the final data for the shape of the sand and soil.)

[0394] That is, the object shape acquisition unit 302B generates the final sand shape data by the following formulas (23) and (24).

[0395] [Formula 16] In addition, for the grid i that the sensor 40 can measure after the excavator 100 performs this excavation action, the data of the sand shape determined in step S110 based on the reasoning result of function g can also be used as the final data of the sand shape.

[0396] If the processing in step S110 is completed, the object shape acquisition unit 302B proceeds to step S112.

[0397] In step S112, the object shape acquisition unit 302B outputs the data of the sand shape generated in step S110 (the height h of the sand). e k and uncertainty s e k ).

[0398] Thus, for example, the target track generation unit 302C can generate the target track of the working part of the excavator 100, namely the tip of the bucket 6, based on the sand shape data output from the object shape acquisition unit 302B.

[0399] Furthermore, for example, the display processing unit 302E can generate an image representing the shape of the sand in the observation area TA surrounding the excavator 100 based on the sand shape data output from the object shape acquisition unit 302B. Therefore, the display processing unit 302E can display the image representing the shape of the sand in the observation area TA surrounding the excavator 100 on the display device 50A, or transmit it via the communication device 60 to the remote operation support device 400 or the remote monitoring support device, and display it on their respective display devices. Furthermore, the display processing unit 302E can also handle uncertainty s. e k This is reflected in an image representing the shape of the sand in the observation area TA surrounding the excavator 100. For example, the display processing unit 302E represents the uncertainty s of each grid i within the observation area TA surrounding the excavator 100 by using the color of the part corresponding to the grid i in the image representing the sand shape. e k Therefore, the display processing unit 302E can display an image representing the shape of the sand in the observation area TA surrounding the excavator 100 in a manner that can identify the difference in uncertainty of each cell i within the observation area TA surrounding the excavator 100.

[0400] In addition to outputting data on the shape of the sand, the object shape acquisition unit 302B can also output other data obtained during the processing of this flowchart. For example, the object shape acquisition unit 302B can output measurement feasibility information m. l k Therefore, the controller 30 can distinguish between grids i in the observation area TA where the sensor 40 can measure the shape of the sand and grids i in the observation area TA where the sensor 40 cannot measure the shape of the sand. Thus, for example, when the display processing unit 302E displays an image representing the shape of the sand in the observation area TA on the display device 50A, it can distinguish between grids i that reflect the output of the sensor 40 and grids that do not reflect the output of the sensor 40.

[0401] If step S114 is completed, the object shape acquisition unit 302B ends the processing of this flowchart.

[0402] Thus, in this example, the object shape acquisition unit 302B can use function g to estimate the shape of the sand after the excavation action based on the shape of the sand before the excavation action, taking into account the trajectory of the bucket 6 during the excavation action of the excavator 100, the excavation reaction force, or the characteristics of the sand.

[0403] Furthermore, in this example, for the case where the predetermined action is a soil-discharging action, the object shape acquisition unit 302B can also generate and output data on the shape of the sand and soil after the soil-discharging action of the excavator 100 through the same processing as described above. In this case, for example, other information ωk is represented by the following equation (25).

[0404] [Equation 17] As shown in equation (25), other information ω k This includes the weight W of the sand and soil contained in the bucket 6 before the excavator 100 performs the soil discharge action. k-1 and volume V k-1 And the weight W of the sand and soil contained in bucket 6 after the soil dumping action. k and volume V k Furthermore, as shown in equation (25), when the spoil heap is a truck bed, other information ω k This can include the positions ρ of the four sides of the truck bed when viewed from above. Therefore, the object shape acquisition unit 302B can estimate the shape of the sand in the truck bed, taking into account the shape of the truck bed.

[0405] Furthermore, for operations performed by a combination of digging and discharging actions of the excavator 100, the object shape acquisition unit 302B can also generate and output data on the shape of the sand after the digging action or the discharging action of the excavator 100 through the same processing as described above.

[0406] Specific examples of methods for generating models after learning. A specific example illustrating the generation method of the learned model LM1 is provided.

[0407] In this example, regarding the above Figure 16 The method for generating the function g corresponding to the learned model LM1 is explained in the processing.

[0408] The training data cj (j=1~L) of the training dataset D are represented by, for example, the following equation (26).

[0409] [Formula 18] Training data c j The input data is h^ e j ,s^ e j h l j s l j m l j bj , f j , λ j κ j ω j The height h of the sand after the excavator 100's digging action, which serves as correct data. r j The combination of input data h^ e j ,s^ e j h l j s l j m l j b j , f j , λ j κ j ω j The input data (h) of the function g in equation (21) e k-1 s e k-1 h l k s l k m l k b k , f k , λ k κ k ω k )correspond.

[0410] As shown in equation (27), the function g has a parameter W, which is optimized by the training dataset D for machine learning.

[0411] [Formula 19] For example, the parameter W is optimized by minimizing the loss function E(W) in the following equation (28), thereby generating the function g corresponding to the learned model LM1.

[0412] [Formula 20] As described above, the training dataset D can be generated based on the logs acquired by the log acquisition unit 2001, the logs acquired by the simulator unit 2002, or a combination of both.

[0413] As described above, in the simulator section 2002, for example, particle simulation such as DEM is used, and the height h of the sand is obtained by ray tracing of a shape sensor such as LIDAR, which is virtually configured relative to the position of the particles. r j .

[0414] Furthermore, as described above, the training dataset D may include a base training dataset generated from logs acquired by the simulator unit 2002 and a micro-call training dataset generated from logs acquired by the log acquisition unit 2001. In this case, the amount of training data contained in the micro-call training dataset can be relatively small.

[0415] With input data h l j S l j m l j Regarding the acquisition of related data, a portion of the measurement data from the sensor group 300 or the measurement data from the virtual shape sensor configured in the virtual space of the simulator unit 2002 can be virtually masked. Therefore, even if no occlusion occurs in the sensor group 300 or the virtual shape sensor configured in the virtual space of the simulator unit 2002, measurement data with virtual occlusion can be acquired. Furthermore, the occlusion region in the observation area TA can be specifically calculated through ray tracing of the shape sensor configured in the virtual space of the simulator unit 2002. Moreover, the occlusion region can be changed while changing the position of the shape sensor configured in the virtual space of the simulator unit 2002. Thus, robust machine learning for the function g can be achieved.

[0416] Furthermore, the input data h^ e j ,s^ e j The output (inference result) of the function g from the previous iteration can be obtained through multiple mining operations repeated sequentially. Furthermore, from the perspective of suppressing the increase in the number of data points, this can be achieved by virtually mixing in large amounts of noise while learning. Therefore, more robust machine learning can be implemented for the function g.

[0417] Thus, the information processing device 200 is able to generate data containing training data c. j The training dataset D is used to generate a function g that is equivalent to the learned model LM1 through machine learning based on the training dataset D.

[0418] <Example 2> Figure 22 This is a flowchart illustrating, schematically, a second example of a process for obtaining data representing the shape of a work object.

[0419] In this example, the third example of the functional structure of the aforementioned operation support system SYS ( Figure 8 ) or the second example of the functional structure of the aforementioned operation support system SYS ( Figure 7 Based on the example of the variation of the above, the explanation is given under the premise that the predetermined action of the excavator 100 is the digging action.

[0420] Figure 22 The flowchart is, for example, performed after the excavator 100 performs the digging action. Figure 22 The flowchart is equivalent to the above Figure 15 Step S15 processing or Figure 17 The processing of step S35. Hereinafter, in this flowchart, it will be explained on the premise of obtaining data representing the shape (sand shape) of the work object after the excavator 100 performs the k-th excavation action from the start of the excavation operation.

[0421] In this example, similar to the first example above, the observation area TA around the excavator 100 is divided into a predetermined number N grids (see reference). Figure 20 Furthermore, in this example, the data representing the shape of the work object after the k-th excavation action (sand shape data) is determined by the height h of the sand. e k Explanation is given under the premise of indication.

[0422] like Figure 22 As shown, in step S202, the controller 30 acquires data representing the track x of the working part during each digging action of the excavator 100. For example, when k=1, the controller 30 acquires the track x of the working part during the first (1st) digging action of the excavator 100. 1 The data. And, for example, when k ≥ 2, the controller 30 acquires the trajectory x of the working part of the excavator 100 during each excavation action from the 1st to the kth time. 1 ~x k The data.

[0423] For example, the controller 30 acquires the target track data generated by the target track generation unit 302C, representing the working part of the excavator 100 during the m-th (1≤m≤k) digging action, and uses it as the track x representing the m-th digging action of the excavator 100. m Furthermore, the controller 30 can also acquire measurement data of the track (trajectory) of the working area based on the outputs of sensors S1 to S5, and use this data as the track x representing the m-th (1≤m≤k) digging action of the excavator 100. m The data.

[0424] Furthermore, during the period from when the excavator 100 begins digging until the digging operation is completed, the result of the processing in step S202 is stored, for example, in the auxiliary storage device 30A or the memory device 30B of the controller 30. Therefore, in step S202 after the k-th digging action, the controller 30 can easily and quickly obtain the track x using the result of the processing in step S202 after the previous digging action. 1 ~x k The data.

[0425] If step S202 is completed, the controller 30 proceeds to step S204.

[0426] In step S204, the object shape acquisition unit 302B of the controller 30 acquires the measurement result of the reaction force of the excavator 100 on the working part during each digging action based on the output of sensors S7 to S9 (measuring reaction force). o The timing data. For example, when k=1, the object shape acquisition unit 302B acquires the timing measurement reaction force F during the first (1st) digging action of the excavator 100. o 1 The data. Furthermore, for example, when k≥2, the object shape acquisition unit 302B acquires the timing measurement reaction force F of each digging action of the excavator 100 from the 1st to the kth time. o 1 ~F o k The data.

[0427] Furthermore, during the period from when the excavator 100 begins digging until the digging operation is completed, the results of the processing in step S204 are stored, for example, in the auxiliary storage device 30A or the memory device 30B of the controller 30. Therefore, in step S204 after the k-th digging operation, the object shape acquisition unit 302B can easily and quickly acquire the measured reaction force F using the results of the processing in step S204 after the previous digging operation. o 1 ~F o k The data.

[0428] If step S204 is completed, the controller 30 proceeds to step S206.

[0429] In step S206, the reaction force estimation unit 302F of the controller 30 acquires the estimated reaction force (estimated reaction force) F of the excavator 100 towards the working site during each excavation action. eThe timing data. For example, when k=1, the object shape acquisition unit 302B acquires the timing measurement reaction force F during the first (1st) digging action of the excavator 100. e 1 The data. Furthermore, for example, when k≥2, the object shape acquisition unit 302B acquires the timing measurement reaction force F of each digging action of the excavator 100 from the 1st to the kth time. e 1 ~F e k The data.

[0430] For example, the reaction force estimation unit 302F uses a function g equivalent to the learned model LM3 to obtain the estimated reaction force F for the mth time through the following equation (29). e m Time series data.

[0431] [Equation 21] Furthermore, the reaction force estimation unit 302F can also import the variable γ representing the characteristics of the sand and soil of the work object, and obtain the estimated reaction force F for the m-th time through the following equation (30). e m Time series data.

[0432] [Equation 22] Therefore, it is possible to improve the estimation of the reaction force F. e m The accuracy.

[0433] If step S206 is completed, the controller 30 proceeds to step S208.

[0434] In step S208, the object shape acquisition unit 302B calculates the measured reaction force F acquired in steps S202 and S204. o m With the estimated reaction force F e m The difference between the estimates (estimated error) is used to correct the initial shape data of the work object. The initial shape is as described above. Figure 15 Step S11 processing or Figure 17 In step S31, the data is set to be based on the height h of the sand. e 0 Specifically, the object shape acquisition unit 302B corrects the initial shape of the object (height h of the sand). e 0 This minimizes the estimation error.

[0435] For example, in the case of k=1, i.e., after the first (first) excavation operation, the object shape acquisition unit 302B acquires the corrected initial shape (height h of the sand) by using the following formula (31) based on the above formula (29). ～ e 0 (Data).

[0436] [Equation 23] Furthermore, for example, in the case where k≥2, i.e., when the excavation action is performed for the second time or later, the object shape acquisition unit 302B acquires the corrected initial shape (height h of the sand) using the following formula (32) based on the above formula (29). ～ e 0 (Data).

[0437] [Equation 24] For example, the object shape acquisition unit 302B uses a function r, which is equivalent to the learned model LM1, to calculate the shape of the object (height h of the sand) after the m-th digging action by the following formula (33). e m (Data).

[0438] [Equation 25] Furthermore, similarly to the case of function g, the object shape acquisition unit 302B can import a variable γ representing the characteristics of the sand in the object, and calculate the shape of the object (height h of the sand) after the m-th excavation action using the following formula (34). e m (Data).

[0439] [Equation 26] This allows for improvement in the shape of the work object (the height h of the sand). e m (The accuracy of )

[0440] If step S208 is completed, the controller 30 proceeds to step S210.

[0441] In step S210, the accuracy estimation unit 302G of the controller 30 estimates the accuracy of the initial shape of the work object that has been corrected in the processing of step S208.

[0442] For example, the accuracy estimation unit 302G uses a function w equivalent to the learned model LM4 to estimate the accuracy c of the initial shape of the work object after correction following the k-th digging action by the following equation (35). k .

[0443] [Equation 27] Additionally, orbit x 1：k This represents the trajectory of the working part during the excavation actions from the 1st to the kth, and measures the reaction force F. o 1：k This represents the measurement results of the reaction force on the working part during the excavation actions from the 1st to the kth.

[0444] Furthermore, similarly to the cases of functions g and r, the object shape acquisition unit 302B can import a variable γ representing the characteristics of the sand in the object, and estimate the accuracy c of the initial shape of the object after correction following the k-th excavation operation using the following equation (36). k .

[0445] [Equation 28] This improves the accuracy c of the initial shape of the work object after correction following the k-th digging action. k The accuracy of the estimation.

[0446] If step S210 is completed, proceed to step S212.

[0447] In step S212, the object shape acquisition unit 302B estimates the latest shape of the object after the kth excavation action and acquires data representing that shape (sand shape data).

[0448] Specifically, the object shape acquisition unit 302B uses the initial shape of the object (height h of the sand) that has been corrected in step S208. ～ e 0 Starting from x0, the shape of the latest work object is estimated based on the trajectory x0 to xk from the first to the kth time.

[0449] For example, when k=1, the object shape acquisition unit 302B uses a function r equivalent to the learned model LM1 to acquire the shape of the object after the first excavation action (the height h of the sand) using the following equation (37). e 1 (Data).

[0450] [Equation 29] Furthermore, when k≥2, the object shape acquisition unit 302B uses a function r equivalent to the learned model LM1 to acquire the shape of the object after the kth digging action (the height h of the sand) using the following equation (38). e k (Data).

[0451] [Formula 30] If step S212 is completed, the controller 30 will end the current flowchart processing.

[0452] Thus, in this example, the controller 30 can correct the initial shape of the work object based on the difference (estimation error) between the estimated reaction force and the measured reaction force, and estimate the latest shape of the work object after the excavator 100 performs the digging action, starting from the corrected initial shape of the work object.

[0453] <Example 3> Figure 23 This is a flowchart illustrating, schematically, a third example of a process for obtaining data representing the shape of a work object.

[0454] In this example, the first example of the functional structure of the aforementioned operation support system SYS is used. Figure 6 ), Example 2 ( Figure 7 ), Example 3 ( Figure 8 This explanation is based on the premise that the predetermined action of the excavator 100 is the soil discharge action, or a variation of the second example.

[0455] Figure 23 The flowchart, for example, is performed after the excavator 100 performs the soil removal action. Specifically, Figure 23 The flowchart is equivalent to the above Figure 16 Step S24 processing or Figure 18 The processing of step S45. Hereinafter, in this flowchart, it will be explained on the premise of obtaining data representing the shape (sand shape) of the work object after the excavator 100 performs the kth soil discharge action from the start of the loading operation.

[0456] In this example, similar to the first or second example above, the explanation is based on the premise that the observation area TA around the excavator 100, i.e., the truck bed, is divided into a predetermined number N grids (see reference). Figure 20 Furthermore, in this example, the data representing the shape of the work object after the k-th digging action (sand shape data) is determined by the height b of the sand relative to the bottom of the truck bed. ek Explanation is given under the premise of indication.

[0457] like Figure 23 As shown, in step S302, the controller 30 estimates the weight w of the sand contained in the bucket 6 before the k-th soil discharge action based on the outputs of sensors S7 to S9. k .

[0458] If step S302 is completed, the controller 30 proceeds to step S304.

[0459] In step S304, the controller 30 acquires the track x of the working part representing the k-th soil discharge action of the excavator 100. k The data.

[0460] For example, the controller 30 acquires the target track data generated by the target track generation unit 302C, representing the working position of the excavator 100 during the k-th soil discharge operation, as the track x representing the working position of the excavator 100 during the k-th soil discharge operation. k The data. Furthermore, the controller 30 can also acquire measurement data of the track (trajectory) of the working part during the k-th earthmoving action of the excavator 100 based on the outputs of sensors S1 to S5, and use this data as a representation of the track x of the working part during the k-th earthmoving action. k The data.

[0461] If step S304 is completed, the controller 30 proceeds to step S306.

[0462] In step S306, the object shape acquisition unit 302B of the controller 30 estimates the latest shape of the object after the excavator 100 performs the kth soil discharge action, namely the shape of the sand in the truck bed, and acquires data representing that shape.

[0463] For example, the object shape acquisition unit 302B uses a function a, which is equivalent to the learned model LM1, to acquire the shape (height b) of the sand in the truck bed after the excavator 100 performs the k-th excavation action, using the following formula (39). e k (Data).

[0464] [Equation 31] Furthermore, the object shape acquisition unit 302B can also import a variable γ representing the characteristics of the sand, and obtain the shape of the sand in the truck bed after the excavator 100 performs the k-th excavation action (the height b of the sand) by the following formula (40). e k (Data).

[0465] [Equation 32] This improves the accuracy of estimating the shape of the sand in the truck bed after the excavator 100 performs the k-th soil discharge action.

[0466] If step S306 is completed, controller 30 will end the current flowchart processing.

[0467] Thus, in this example, the controller 30 can estimate the latest shape of the work object after the excavator 100 performs the soil removal action based on the shape of the work object before the excavator 100 performs the soil removal action and the trajectory of the work part when the excavator 100 performs the soil removal action.

[0468] [effect] Next, the functions of the construction machinery, information processing device, and program involved in this embodiment will be explained.

[0469] In the first embodiment of this invention, the construction machinery includes a processing device. The construction machinery is, for example, the excavator 100 described above. The processing device is, for example, the controller 30 described above. Specifically, the processing device acquires data representing the state of the construction machinery and related to the shape of the work object in response to the operation of the construction machinery, and estimates the shape of the work object after the operation of the construction machinery based on the acquired data.

[0470] Furthermore, in the first embodiment of this invention, the information processing device can acquire data representing the state of the construction machinery and related to the shape of the work object in response to the operation of the construction machinery, and estimate the shape of the work object of the construction machinery after the operation of the construction machinery based on the acquired data. The information processing device is, for example, the controller 30, the information processing device 200, or the remote operation support device 400 described above.

[0471] Furthermore, in the first aspect of this embodiment, the program can cause the information processing device to execute the first step and the second step. Specifically, in the first step, data representing the state of the construction machinery and related to the shape of the work object can be acquired in accordance with the operation of the construction machinery. Then, in the second step, the shape of the work object of the construction machinery can be estimated based on the data acquired in the first step.

[0472] Therefore, construction machinery or information processing devices (hereinafter referred to as "construction machinery, etc.") can, for example, estimate the state of the work object by considering changes in the shape of the work object corresponding to the actions of the construction machinery. Thus, construction machinery, etc., can more accurately grasp the shape of the work object.

[0473] Furthermore, in the second embodiment of this invention, based on the first embodiment described above, the construction machinery may be equipped with a first acquisition device. The first acquisition device is, for example, the aforementioned sensors S1 to S3. Specifically, the first acquisition device can acquire data related to the track of the working part of the construction machinery. Then, a processing device or information processing device (hereinafter referred to as "processing device, etc.") can estimate the shape of the work object based on the data related to the track of the working part when performing a predetermined action.

[0474] Therefore, construction machinery can estimate the state of the work object by considering the changes in the shape of the work object, based on the track of the work area and the positional relationship between the work area and the work object.

[0475] Furthermore, in the third embodiment of this invention, based on the second embodiment described above, the construction machinery may be equipped with a second acquisition device. The second acquisition device is, for example, the aforementioned sensors S7 to S9. Specifically, the second acquisition device can acquire data related to the reaction force from the work object to the work site or data related to the weight of the sand held by the work site. Then, the processing device or the like can estimate the shape of the work object based on the data related to the reaction force to the work site when performing a predetermined action or the data related to the weight of the sand held by the work site.

[0476] Therefore, construction machinery can, for example, identify contact with underground rocks or the hardness of sand based on the reaction force exerted on the work area when the machinery performs a predetermined action. Furthermore, construction machinery can, for example, identify the amount of sand discharged into the work object based on the weight of the sand held in the bucket, which serves as the work area. Thus, construction machinery can take these identification results into account to more accurately estimate the shape of the work object.

[0477] Furthermore, in the fourth embodiment of this invention, based on the second embodiment described above, the construction machinery can be equipped with a measuring device. The measuring device is, for example, the sensor 40 described above. Specifically, the measuring device can measure the shape of the work object surrounding the construction machinery. Then, the processing device or the like can estimate the shape of the work object based on the measurement data of the work object's shape and data related to the trajectory of the work area when performing a predetermined action.

[0478] Therefore, construction machinery, for example, can take the shape of the work object at a certain point in time as a starting point, and estimate the shape of the work object based on the track of the work area and considering the changes in the shape of the work object.

[0479] Furthermore, in the fifth aspect of this embodiment, based on the fourth aspect described above, the processing device or the like can estimate the shape of the work object after performing the predetermined action based on measurement data of the shape of the work object before performing the predetermined action, measurement data of the shape of the work object after performing the predetermined action, and data related to the track of the work part when performing the predetermined action.

[0480] Therefore, the shape of the work object after the construction machinery performs a predetermined action can be estimated based on the measurement data of the measuring device, and for parts that cannot be measured due to obstruction, the shape of the work object can be estimated based on the track of the work area.

[0481] Furthermore, in the sixth embodiment of this invention, based on the fourth or fifth embodiment described above, the processing device or the like can obtain data representing the shape of the work object based on the estimation result of the shape of the work object and the measurement data of the shape of the work object obtained by the measuring device after the estimation result is output.

[0482] Thus, the processing device, for example, can use the results of machine learning, based on the difference between the estimated shape of the work object and the measurement results of the measuring device after outputting the estimated shape, to more accurately estimate the data representing the shape of the work object.

[0483] Furthermore, in the seventh embodiment of this invention, based on the third embodiment described above, the processing device or the like can estimate the reaction force from the work object to the work site when the predetermined action is performed, based on data representing the shape of the work object before the construction machinery performs the predetermined action and data related to the track of the work site when the construction machinery performs the predetermined action. Furthermore, the processing device or the like can also correct the data representing the shape of the work object before the construction machinery performs the predetermined action, based on the estimated reaction force from the work object to the work site when the construction machinery performs the predetermined action and data related to the actual reaction force from the work object to the work site when the construction machinery performs the predetermined action. Moreover, the processing device or the like can also estimate the shape of the work object after the predetermined action is performed, based on the corrected data representing the shape of the work object before the construction machinery performs the predetermined action and data related to the track of the work site when the construction machinery performs the predetermined action.

[0484] Therefore, even if the accuracy of the data representing the shape of the work object before the construction machinery performs the predetermined action is relatively low, the processing device or the like can correct the data based on the estimation error of the estimation result of the reaction force when the construction machinery performs the predetermined action. Thus, the processing device or the like can estimate the shape of the work object after the construction machinery performs the predetermined action with relatively high accuracy.

[0485] Furthermore, in the eighth embodiment of this invention, based on the seventh embodiment described above, the processing device or the like can correct the data representing the initial shape of the work object based on the estimated reaction force from the work object to the work site during each predetermined action performed by the construction machinery, starting from the initial shape of the work object, and data related to the actual reaction force from the work object to the work site during the predetermined action. Moreover, the processing device or the like can also estimate the shape of the work object after the construction machinery has performed multiple predetermined actions based on the data representing the corrected initial shape of the work object and data related to the track of the work site during each predetermined action performed by the construction machinery.

[0486] Therefore, even if the accuracy of the data representing the initial shape of the construction machinery is relatively low, the processing device can correct the data based on the estimation error of the estimated reaction force when the construction machinery performs the predetermined action. Thus, the processing device can estimate the shape of the work object after the construction machinery performs the predetermined action with relatively high accuracy.

[0487] Furthermore, in the ninth embodiment of this invention, based on the seventh or eighth embodiment described above, the processing device or the like can limit at least one of the operating speed and operating range of the construction machinery's predetermined action based on data representing the uncertainty of the shape of the work object corresponding to the position within the work object. This data refers to data representing the shape of the work object before the construction machinery performs the predetermined action.

[0488] Therefore, for example, when the uncertainty of the shape data representing the shape of the work object before the construction machinery performs a predetermined action is relatively high, the processing device or the like can slow down the speed of the predetermined action of the construction machinery or reduce the range of action. Thus, even when the accuracy of the data representing the shape of the work object before the construction machinery performs a predetermined action is relatively low, when the construction machinery performs a predetermined action based on that data, it is possible to suppress the reaction force from the work object to the working part of the construction machinery from becoming too large.

[0489] Furthermore, in the 10th embodiment of this invention, based on any of the 1st to 9th embodiments described above, the processing device or information processing device can estimate the shape of the work object based on the characteristics of the sand in the work object or the sand added to the work object in response to the operation of the construction machinery.

[0490] As a result, construction machinery can take into account the characteristics of sand and soil to more accurately estimate the shape of the work object.

[0491] Furthermore, in the 11th embodiment of this invention, based on the 10th embodiment described above, the characteristics of the sand may include at least one of the angle of repose of the sand, the moisture content of the sand, and the particle size of the sand.

[0492] Therefore, construction machinery can take into account the angle of repose of the sand, the moisture content of the sand, or the particle size of the sand to more accurately estimate the shape of the work object.

[0493] Furthermore, in the 12th embodiment of this invention, based on any of the 1st to 11th embodiments described above, there can be multiple predetermined actions of the construction machinery. Moreover, the processing device or information processing device can estimate the shape of the work object based on the predetermined actions performed by the construction machinery among the multiple predetermined actions.

[0494] Therefore, construction machinery and other equipment can estimate the state of the work object based on the predetermined actions performed by the construction machinery.

[0495] Furthermore, in the 13th embodiment of this invention, based on any of the 2nd to 9th embodiments described above, the working part of the construction machinery can be a bucket. Moreover, the predetermined action of the construction machinery can be a digging action or a soil removal action.

[0496] Therefore, construction machinery can take into account the changes in the shape of the work object corresponding to the digging or dumping actions of the construction machinery to more accurately estimate the shape of the sand and soil of the work object.

[0497] Furthermore, in the 14th embodiment of this invention, based on any of the 1st to 13th embodiments described above, the construction machinery may include a control device that controls the operation of the construction machinery based on an estimation of the shape of the work object. Additionally, the information processing device may include a control unit that controls the operation of the construction machinery based on the estimation of the shape of the work object.

[0498] Therefore, construction machinery can be controlled to move according to the shape of the work object.

[0499] Furthermore, in the 15th embodiment of this invention, based on any of the 1st to 14th embodiments described above, the construction machinery may be equipped with a display device that displays an image representing the shape of the work object based on an estimation result of the shape of the work object. The display device is, for example, the output device 50 described above.

[0500] Furthermore, in the 15th embodiment of this invention, based on any of the 1st to 14th embodiments described above, the program can cause the support device to execute steps 1 to 3. Specifically, in the 1st step, data representing the state of the construction machinery and related to the shape of the work object can be acquired in accordance with the operation of the construction machinery. In the 2nd step, the shape of the work object of the construction machinery can be estimated based on the data acquired in the 1st step. In the 3rd step, an image representing the shape of the work object can be displayed on the display unit based on the estimated shape of the work object. The support device is, for example, a remote operation support device 400.

[0501] Therefore, construction machinery and other equipment can provide operators with estimates of the shape of the work object.

[0502] Furthermore, in the 16th embodiment of this invention, based on the 15th embodiment described above, the processing device or the like can perform the following processing: estimating the shape of the work object after the construction machinery operates based on the operation of the construction machinery; acquiring data representing the shape of the work object after the construction machinery operates based on the estimation result of the shape of the work object; and acquiring data representing the uncertainty of the shape of the work object corresponding to its position within the work object, based on the acquired data. Then, the display device can display an image representing the shape of the work object in a manner that can identify the difference in uncertainty corresponding to the position within the work object.

[0503] Furthermore, in the 16th embodiment of this invention, based on the 15th embodiment described above, the program can cause the support device to perform a second step and a fourth step. In the second step, the shape of the work object after the construction machinery operates is estimated based on the operation of the construction machinery, and data representing the shape of the work object after the operation is obtained based on the estimated shape of the work object. In the fourth step, data representing the uncertainty of the estimated shape of the work object corresponding to its position within the work object, obtained in the second step, is obtained. Moreover, the program can cause the support device to perform a third step, in which an image representing the estimated shape of the work object is displayed on a display unit in a manner that can identify the difference in uncertainty corresponding to the position within the work object.

[0504] Thus, construction machinery and other equipment enable users to visually identify the estimated shape of the work object, and at the same time enable users to identify the difference in uncertainty of the estimated result corresponding to the position within the observation range of the object.

[0505] The embodiments have been described in detail above, but the present invention is not limited to this specific embodiment. Various modifications and alterations can be made within the scope of the spirit described in the technical solution.

[0506] Finally, this application claims priority based on Japanese Patent Application No. 2023-200126, filed on November 27, 2023, and the entire contents of the Japanese Patent Application are incorporated herein by reference.

[0507] Symbol Explanation 1-Lower traveling body, 3-Upper slewing body, 4-Boom, 5-Stick, 6-Bucket, 30-Controller, 31-Hydraulic control valve, 32-Shuttle valve, 33-Hydraulic control valve, 40-Sensor, 40B-Sensor, 40F-Sensor, 40L-Sensor, 40R-Sensor, 50-Output device, 50A-Display device, 52-Input device, 60-Communication device, 100-Excavator, 150-Support device, 200-Information processing device, 300-Sensor group, 300-1~300-M-Sensors, 301-Motion log providing unit, 301A-Motion log recording unit, 301B-Motion log storage unit, 301C-Motion log sending unit, 302-Operation support unit, 302A-Learning unit Completed Model Storage Unit, 302B-Work Object Shape Acquisition Unit, 302C-Target Track Generation Unit, 302D-Motion Control Unit, 302E-Display Processing Unit, 302F-Reaction Force Estimation Unit, 302G-Accuracy Estimation Unit, 400-Remote Operation Support Device, 2001-Log Acquisition Unit, 2002-Simulator Unit, 2003-Log Storage Unit, 2004-Training Data Generation Unit, 2004A~2004D-Training Data Generation Unit, 2005-Machine Learning Unit, 2005A~2005D-Machine Learning Unit, 2006-Completed Learning Model Storage Unit, 2007-Distribution Unit, AT-Accessories, LM1~LM4-Completed Learning Model, S1~S9-Sensors, SYS-Operation Support System.

Claims

1. A construction machine, wherein, The construction machinery is equipped with a processing device that acquires data representing the state of the construction machinery and related to the shape of the work object in response to the actions of the construction machinery, and estimates the shape of the work object after the actions of the construction machinery based on the acquired data.

2. The construction machinery according to claim 1, wherein, The construction machinery is equipped with a first acquisition device, which acquires data related to the track of the working part of the construction machinery. The processing device estimates the shape of the work object based on data related to the track of the work area when performing a predetermined action.

3. The construction machinery according to claim 2, wherein, The construction machinery is equipped with a second acquisition device, which acquires data related to the reaction force from the work object to the work site or data related to the weight of sand held by the work site. The processing device estimates the shape of the work object based on data related to the reaction force exerted on the work site when the predetermined action is performed or data related to the weight of the sand held by the work site.

4. The construction machinery according to claim 2, wherein, The construction machinery is equipped with a measuring device that measures the shape of the work object surrounding the construction machinery. The processing device estimates the shape of the work object based on measurement data of the shape of the work object and data related to the trajectory of the work part when performing the predetermined action.

5. The construction machinery according to claim 4, wherein, The processing device estimates the shape of the work object after performing the predetermined action based on measurement data of the shape of the work object before performing the predetermined action, measurement data of the shape of the work object after performing the predetermined action, and data related to the track of the work part when performing the predetermined action.

6. The construction machinery according to claim 4 or 5, wherein, The processing device acquires data representing the shape of the work object based on the estimated shape of the work object and the measurement data of the shape of the work object obtained by the measuring device after the estimated shape is output.

7. The construction machinery according to claim 3, wherein, The processing device performs the following processing: The reaction force from the work object to the work part when the predetermined action is performed is estimated based on data representing the shape of the work object before the predetermined action is performed and data related to the trajectory of the work part when the predetermined action is performed. The data representing the shape of the work object before performing the predetermined action are corrected based on the estimated reaction force from the work object to the work location when the predetermined action is performed and the data related to the actual reaction force from the work object to the work location when the predetermined action is performed. and The shape of the work object after performing the predetermined action is estimated based on corrected data representing the shape of the work object before performing the predetermined action and data related to the trajectory of the work part when performing the predetermined action.

8. The construction machinery according to claim 7, wherein, The processing device performs the following processing: The data representing the initial shape of the work object are corrected based on the estimated reaction force from the work object to the work site during each of the multiple predetermined actions performed starting from the initial shape of the work object, and the data related to the actual reaction force from the work object to the work site during the execution of the predetermined actions. and The shape of the work object after performing multiple predetermined actions is estimated based on data representing the initial shape of the work object after correction and data related to the trajectory of the work part when performing the predetermined actions in each of the multiple predetermined actions.

9. The construction machinery according to claim 7 or 8, wherein, The processing device limits at least one of the speed and range of the predetermined action based on data representing the uncertainty of the shape of the work object corresponding to its position within the work object, the data being data representing the shape of the work object before the predetermined action is performed.

10. The construction machinery according to any one of claims 1 to 5, 7 and 8, wherein, The processing device estimates the shape of the work object based on the characteristics of the sand or soil added to the work object based on the actions of the construction machinery.

11. The construction machinery according to claim 10, wherein, The properties include at least one of the following: the angle of repose of the sand, the moisture content of the sand, and the particle size of the sand.

12. The construction machinery according to any one of claims 2 to 5, 7 and 8, wherein, There are multiple predetermined actions, The processing device estimates the shape of the work object based on the predetermined actions performed by the construction machinery among a plurality of predetermined actions.

13. The construction machinery according to any one of claims 2 to 5, 7 and 8, wherein, The work area is the bucket. The predetermined action is either an excavation action or a soil removal action.

14. The construction machinery according to any one of claims 1 to 5, 7 and 8, wherein, The construction machinery is equipped with a control device, which controls the movement of the construction machinery based on the estimated shape of the work object.

15. The construction machinery according to any one of claims 1 to 5, 7 and 8, wherein, The construction machinery is equipped with a display device that displays an image representing the shape of the work object based on an estimation of the object's shape.

16. The construction machinery according to claim 15, wherein, The processing device estimates the shape of the work object after the construction machinery's movement in response to the machinery's actions. Based on the estimated shape, it acquires data representing the shape of the work object after the machinery's movement, and also acquires data representing the uncertainty of the work object's shape corresponding to its position within the work object, based on the acquired data. The display device displays an image representing the shape of the work object in a manner that enables it to identify the difference in uncertainty corresponding to the position within the work object.

17. An information processing apparatus, wherein, The information processing device acquires data representing the state of the construction machinery and related to changes in the shape of the work object in response to the actions of the construction machinery, and estimates the shape of the work object after the actions of the construction machinery based on the acquired data.

18. A program that causes an information processing device to perform the following steps: The steps of acquiring data representing the state of the construction machinery in relation to changes in the shape of the work object being worked on, corresponding to the actions of the construction machinery; and The step of estimating the shape of the work object after the construction machinery has been operated based on the acquired data.

19. A program that causes a support device to perform the following steps: A step of acquiring data representing the state of the construction machinery in relation to changes in the shape of the object being worked on, corresponding to the actions of the construction machinery; The steps of estimating the shape of the work object after the construction machinery has moved based on the acquired data; and The step of displaying the shape of the work object on the display unit based on the estimated shape of the work object.

Images