System for work machine, and excavator

Abstract

To provide a system for work machines capable of improving work efficiency.SOLUTION: A system for a work machine relevant to an embodiment of the disclosure includes: a spatial recognition unit that is capable of measuring three-dimensional shape of a target that exists around the work machine; and a controller that is configured to complement the three-dimensional shape of the ground that is not measured by the spatial recognition unit on the basis of measurement information including at least the three-dimensional shape of the ground around the work machine that is measured by the spatial recognition unit.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a system for a working machine and an excavator.

Background Art

[0002] Conventionally, an object detection device is often provided to detect an object existing around a working machine (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Patent Document 1 describes calculating the coordinates of each object detected by an object detection device in a reference coordinate system to grasp the positional relationship between each object such as an obstacle and the excavator 100. However, when the object detection device performs detection, there may be a region that cannot be detected due to the terrain or objects existing around. When a region that cannot be detected by the object detection device occurs, since there is no information regarding the region, it is difficult to perform control on the ground in the region.

[0005] One aspect of the present invention can improve work efficiency by complementing the three-dimensional shape of the ground existing in the region that has not been measured and enabling work on the complemented three-dimensional ground.

Means for Solving the Problems

[0006] A system for a working machine according to one aspect of the present invention includes a spatial recognition device capable of measuring the three-dimensional shape of an object existing around the working machine, and a control device configured to complement the three-dimensional shape of the ground that has not been measured by the spatial recognition device based on measurement information including at least the three-dimensional shape of the ground around the working machine measured by the spatial recognition device.

Effect of the Invention

[0007] According to one aspect of the present invention, the working efficiency is improved by complementing the ground existing in the unmeasured area.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Best Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Also, the embodiments described below are illustrative and do not limit the invention, and not all features and combinations thereof described in the embodiments are necessarily essential to the invention. In each drawing, the same or corresponding components are denoted by the same or corresponding reference numerals, and the description thereof may be omitted.

[0010] Hereinafter, in the embodiments of the present invention, an example in which an excavator is used as an example of a working machine will be described, but the present invention is not limited to excavators. It may be applied to construction machines, standard machines, application machines, forestry machines, or transport machines based on hydraulic excavators.

[0011] (First Embodiment) First, referring to FIG. 1, an overview of the excavator 100 according to the present embodiment will be described. FIG. 1 is a side view of the excavator 100 according to the first embodiment. FIG. 2 is a top view of the excavator 100 according to the first embodiment.

[0012] An upper swing body 3 is rotatably mounted on a lower traveling body 1 of the excavator 100 via a swing mechanism 2. A boom 4 is attached to the upper swing body 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 as an end attachment is attached to the tip of the arm 5. The end attachment may be a slope bucket, a dredging bucket, or the like.

[0013] The boom 4, the arm 5, and the bucket 6 constitute an excavation attachment which is an example of the attachment AT, and are respectively hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9. A boom angle sensor S1 is attached to the boom 4, an arm angle sensor S2 is attached to the arm 5, and a bucket angle sensor S3 is attached to the bucket 6. The excavation attachment may be provided with a bucket tilt mechanism.

[0014] The boom angle sensor S1 detects the rotation angle of the boom 4. In this embodiment, the boom angle sensor S1 is an acceleration sensor and can detect the boom angle, which is the rotation angle of the boom 4 with respect to the upper swing body 3. The boom angle, for example, becomes the minimum angle when the boom 4 is lowered to the lowest position and increases as the boom 4 is raised.

[0015] The boom angle sensor S1 may include, for example, a rotary encoder, an acceleration sensor, a six-axis sensor, an IMU (Inertial Measurement Unit), etc. Further, the boom angle sensor S1 may include a potentiometer using a variable resistor, a cylinder stroke sensor that detects the stroke amount of a hydraulic cylinder (boom cylinder 7) corresponding to the boom angle, etc. The same applies to the arm angle sensor S2, the bucket angle sensor S3, and the body tilt sensor S4 below. The detection signal corresponding to the boom angle by the boom angle sensor S1 is taken into the controller 30.

[0016] The arm angle sensor S2 detects the rotation angle of the arm 5. In this embodiment, the arm angle sensor S2 is an acceleration sensor and can detect the arm angle, which is the rotation angle of the arm 5 with respect to the boom 4. The arm angle, for example, becomes the minimum angle when the arm 5 is closed to the maximum extent and increases as the arm 5 is opened.

[0017] The bucket angle sensor S3 detects the rotation angle of the bucket 6. In this embodiment, the bucket angle sensor S3 is an acceleration sensor and can detect the bucket angle, which is the rotation angle of the bucket 6 with respect to the arm 5. The bucket angle, for example, becomes the minimum angle when the bucket 6 is closed to the maximum extent and increases as the bucket 6 is opened.

[0018] The boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 may be a potentiometer using a variable resistor, a stroke sensor that detects the stroke amount of the corresponding hydraulic cylinder, a rotary encoder that detects the rotation angle around the connecting pin, or the like. The boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 constitute an attitude sensor that detects the attitude of the excavation attachment.

[0019] The upper swing body 3 is provided with a cabin 10 which is a driver's cab and is equipped with a power source such as an engine 11. Further, a machine body inclination sensor S4, a swing angle sensor S5, an imaging device S6, and a space recognition device S7 are attached to the upper swing body 3. Further, a communication device T1 and a positioning device PS are attached to the upper swing body 3.

[0020] The machine body inclination sensor S4 is configured to detect the inclination of the upper swing body 3 with respect to a predetermined plane. In the present embodiment, the machine body inclination sensor S4 is an acceleration sensor that detects the inclination angle around the front-rear axis and the inclination angle around the left-right axis of the upper swing body 3 with respect to the horizontal plane. The front-rear axis and the left-right axis of the upper swing body 3 pass through, for example, a point on the swing axis of the excavator 100 that is the excavator center point and are orthogonal to each other.

[0021] The swing angle sensor S5 is configured to detect the swing angular velocity of the upper swing body 3. In the present embodiment, the swing angle sensor S5 is a gyro sensor. The swing angle sensor S5 may be a resolver or a rotary encoder or the like. The swing angle sensor S5 may detect the swing speed. The swing speed may be calculated from the swing angular velocity.

[0022] In addition, when the machine body inclination sensor S4 includes a gyro sensor, a 6-axis sensor, an IMU, etc. that can detect the angular velocity around three axes, the swing state (for example, the swing angular velocity) of the upper swing body 3 may be detected based on the detection signal of the machine body inclination sensor S4. In this case, the swing angle sensor S5 may be omitted.

[0023] The imaging device S6 is configured to acquire images of the surroundings of the excavator 100. In this embodiment, the imaging device S6 includes a front camera S6F that images the space in front of the excavator 100, a left camera S6L that images the space to the left of the excavator 100, a right camera S6R that images the space to the right of the excavator 100, and a rear camera S6B that images the space behind the excavator 100.

[0024] The imaging device S6 is, for example, a monocular camera having an image sensor such as a CCD or a CMOS, and outputs the captured image to the first display device D3 via the controller 30.

[0025] The input device D2 receives operation inputs from the operator and outputs them to the controller 30. The input device D2 includes, for example, any hardware operation means such as a touch panel, a touch pad, buttons, toggles, rotary knobs, etc. Further, the input device D2 may include software operation means that can be operated through hardware operation means such as virtual button icons on an operation screen displayed on the first display device D3 or the like.

[0026] As shown in FIG. 2, the front camera S6F is attached to, for example, the roof of the cab 10. The left camera S6L is attached to the left end of the upper revolving body 3 on the upper surface. The right camera S6R is attached to the right end of the upper revolving body 3 on the upper surface. The rear camera S6B is attached to the rear end of the upper revolving body 3 on the upper surface.

[0027] Note that the two-dot chain line in FIG. 2 represents the viewing angle (imaging range) of each of the imaging devices S6 in a top view. Specifically, the imaging range CZ includes the imaging range CZB of the rear camera S6B, the imaging range CZF of the front camera S6F, the imaging range CZL of the left camera S6L, and the imaging range CZR of the right camera S6R. The four monocular cameras are preferably attached to the upper revolving body 3 so as not to protrude from the upper surface of the upper revolving body 3 as shown in FIG. 2.

[0028] In this embodiment, by providing the imaging device S6 in the above-described arrangement, it is possible to image an object existing around the excavator 100.

[0029] The space recognition device S7 is configured to recognize the state of the space around the excavator 100. The space recognition device S7 includes a rear space recognition device S7B that detects the space behind the excavator 100, a left space recognition device S7L that detects the space to the left of the excavator 100, a right space recognition device S7R that detects the space to the right of the excavator 100, and a front space recognition device S7F that detects the space in front of the excavator 100.

[0030] As an example, the space recognition device S7 uses LIDAR to measure the three-dimensional shape of an object existing around the excavator 100. LIDAR measures, for example, the distance between more than one million points within the monitoring range and the LIDAR. Note that this embodiment is not limited to the method using LIDAR, and any space recognition device capable of measuring the distance to an object may be used. For example, a stereo camera may be used, or a ranging device such as a distance image camera or a millimeter-wave radar may be used. When a millimeter-wave radar or the like is used as the space recognition device S7, a large number of signals (such as laser light) may be transmitted from the space recognition device S7 toward the object, and the distance and direction of the object may be derived from the reflected signal by receiving the reflected signal.

[0031] The rear space recognition device S7B is attached to the rear end of the upper slewing body 3 on its upper surface. The left space recognition device S7L is attached to the left end of the upper surface of the upper slewing body 3. The right space recognition device S7R is attached to the right end of the upper surface of the upper slewing body 3. The front space recognition device S7F is attached to the front end of the upper surface of the cab 10.

[0032] Note that the dotted lines in Fig. 2 represent the viewing angles (monitoring ranges) of the upper surface views of the respective space recognition devices S7. Specifically, it includes the imaging range OZB of the rear space recognition device S7B, the imaging range OZF of the front space recognition device S7F, the imaging range OZL of the left space recognition device S7L, and the imaging range OZR of the right space recognition device S7R. The four monocular cameras are preferably attached to the upper slewing body 3 so as not to protrude from the upper surface of the upper slewing body 3 as shown in Fig. 2.

[0033] The space recognition device S7 may be configured to detect a predetermined object within a predetermined area set around the excavator 100. For example, the space recognition device S7 may have a person detection function configured to detect a person while distinguishing between a person and an object other than a person.

[0034] The communication device T1 is a device that controls communication between the excavator 100 and the outside. The communication device T1 controls, for example, wireless communication between an external GNSS (Global Navigation Satellite System) surveying system and the excavator 100. The excavator 100 can acquire design data via wireless communication by using the communication device T1. However, the excavator 100 may acquire design data by using a semiconductor memory or the like. Note that the design data includes three-dimensional design data.

[0035] The positioning device PS is configured to acquire information regarding the position of the excavator 100. In the present embodiment, the positioning device PS is configured to measure the position and orientation of the excavator 100. Specifically, the positioning device PS is a GNSS receiver incorporating an electronic compass, and measures the latitude, longitude, and altitude of the current position of the excavator 100, and also measures the orientation of the excavator 100.

[0036] FIG. 3 is a diagram showing a configuration example of the drive control system of the excavator 100 in FIG. 1. In FIG. 3, the mechanical power transmission system is indicated by a double line, the hydraulic line is indicated by a thick solid line, the pilot line is indicated by a broken line, and the electric drive / control system is indicated by a thin solid line.

[0037] The engine 11 is a power source of the excavator 100. In the present embodiment, the engine 11 is a diesel engine that employs isochronous control to maintain a constant engine speed regardless of the increase or decrease of the engine load. The fuel injection amount, fuel injection timing, boost pressure, etc. in the engine 11 are controlled by an engine control unit (ECU) D7.

[0038] The rotating shafts of the main pump 14 and the pilot pump 15, each serving as a hydraulic pump, are connected to the rotating shaft of the engine 11. The control valve unit 17 is connected to the main pump 14 via an operating oil line.

[0039] The control valve unit 17 is a hydraulic control device that controls the hydraulic system of the excavator 100. Hydraulic actuators such as the left and right travel hydraulic motors, the boom cylinder 7, the arm cylinder 8, the bucket cylinder 9, and the swing hydraulic motor are connected to the control valve unit 17 via operating oil lines. The swing hydraulic motor may be a swing electric generator.

[0040] Figure 3 shows the connection relationship between the controller 30, the first display device D3, and the second display device D3S. In the present embodiment, the first display device D3 and the second display device D3S are connected to the controller 30. The first display device D3, the second display device D3S, and the controller 30 may be connected via a communication network such as CAN.

[0041] The space recognition device S7 can measure the three-dimensional shape of an object around the excavator 100 and can detect the presence or absence of an object existing in the space around the excavator 100 and the distance to the object. The space recognition device S7 outputs the result of measuring the space to the controller 30 as measurement information.

[0042] The first display device D3 includes a conversion processing unit D3a that generates an image. In the present embodiment, the conversion processing unit D3a generates a display camera image based on the output of a camera as the imaging device S6. The imaging device S6 is connected to the first display device D3 via, for example, a dedicated line.

[0043] The conversion processing unit D3a generates a display image based on the output of the controller 30. In the present embodiment, the conversion processing unit D3a converts various information output by the controller 30 into an image signal. The information output by the controller 30 includes, for example, data indicating the temperature of the engine cooling water, data indicating the temperature of the hydraulic oil, data indicating the remaining fuel amount, data indicating the remaining amount of urea water, data indicating the position of the working part of the bucket 6, data indicating the orientation of the slope surface of the work target, data indicating the orientation of the excavator 100, data indicating the operation direction for aligning the excavator 100 directly with the slope surface, and the like.

[0044] The second display device D3S includes a conversion processing unit D3Sa that generates an image, similar to the first display device D3. In the present embodiment, the second display device D3S is not directly connected to the imaging device S6. Therefore, the conversion processing unit D3Sa does not generate a camera image. However, the conversion processing unit D3Sa may generate a camera image when the second display device D3S is directly connected to the imaging device S6.

[0045] The conversion processing unit D3Sa generates a display image based on the output of the controller 30. In the present embodiment, the conversion processing unit D3Sa converts various information output by the controller 30 into an image signal.

[0046] The conversion processing unit D3a may be realized as a function of the controller 30 instead of as a function of the first display device D3. The same applies to the conversion processing unit D3Sa. In this case, the imaging device S6 is connected to the controller 30 instead of the first display device D3.

[0047] The first display device D3 and the second display device D3S operate by receiving power supply from the battery 70. The battery 70 is charged with the power generated by the alternator 11a (generator) of the engine 11. The power of the battery 70 is supplied not only to the controller 30, the first display device D3, and the second display device D3S, but also to the electrical components 72 of the excavator 100 and the like. The starter 11b of the engine 11 is driven by the power from the battery 70 to start the engine 11.

[0048] The engine 11 is controlled by an engine control unit D7. Various data indicating the state of the engine 11 are constantly transmitted from the engine control unit D7 to the controller 30. The various data indicating the state of the engine 11 are an example of the operation information of the hydraulic excavator 100, and include, for example, data indicating the coolant water temperature detected by a water temperature sensor 11c as an operation information acquisition unit. The controller 30 accumulates this data in a temporary storage unit (memory) 30a and can transmit it to the first display device D3 when necessary.

[0049] The following various data are supplied to the controller 30 as the operation information of the hydraulic excavator 100 and stored in the temporary storage unit 30a of the controller 30.

[0050] For example, data indicating the swash plate tilt angle is supplied from a regulator 13 of a main pump 14, which is a variable displacement hydraulic pump, to the controller 30. Also, data indicating the discharge pressure of the main pump 14 is supplied from a discharge pressure sensor 14b to the controller 30. These data are stored in the temporary storage unit 30a. Further, an oil temperature sensor 14c is provided in a pipeline between a tank storing the hydraulic oil sucked by the main pump 14 and the main pump 14, and data representing the temperature of the hydraulic oil flowing through the pipeline is supplied from the oil temperature sensor 14c to the controller 30. The regulator 13, the discharge pressure sensor 14b, and the oil temperature sensor 14c are examples of operation information acquisition units.

[0051] Data indicating the fuel storage amount is supplied from a fuel storage amount detection unit 55a in a fuel storage unit 55 to the controller 30. In the present embodiment, data indicating the remaining fuel amount state is supplied from a fuel remaining amount sensor as the fuel storage amount detection unit 55a in a fuel tank as the fuel storage unit 55 to the controller 30.

[0052] Specifically, the fuel remaining amount sensor is composed of a float that follows the liquid level and a variable resistor (potentiometer) that converts the vertical movement amount of the float into a resistance value. With this configuration, the fuel remaining amount sensor can display the fuel remaining amount state continuously on the first display device D3. The detection method of the fuel storage capacity detection unit can be appropriately selected according to the usage environment, etc., and a detection method that can display the fuel remaining amount state stepwise may be adopted. These configurations are the same for the urea water tank as well.

[0053] The operation device 26 is provided near the driver's seat in the cabin 10 and is used for the operator to operate various driven elements. Specifically, the operation device 26 is used for the operator to operate hydraulic actuators such as the left and right traveling hydraulic motors, boom cylinder 7, arm cylinder 8, bucket cylinder 9, and slewing hydraulic motor. As a result, the operator can operate the driven elements driven by the hydraulic actuators. The operation device 26 includes a pedal device and a lever device for operating each driven element.

[0054] The operation sensor 29 is configured to detect the operation content of the operator using the operation device 26. In this embodiment, the operation sensor 29 detects the operation direction and operation amount of the operation device 26 corresponding to each hydraulic actuator, and outputs an electrical signal (hereinafter also referred to as an operation signal) corresponding to the detected value to the controller 30. In this embodiment, the controller 30 controls the opening area of the proportional valve 31 according to the output of the operation sensor 29. Then, the controller 30 supplies the hydraulic oil discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17. The pressure (pilot pressure) of the hydraulic oil supplied to each pilot port is, in principle, a pressure corresponding to the operation direction and operation amount of the operation device 26 corresponding to each hydraulic actuator. In this way, the operation device 26 is configured to be able to supply the hydraulic oil discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17. Thereby, the hydraulic actuator can be driven.

[0055] Further, the direction change valve for driving each hydraulic actuator incorporated in the control valve unit 17 may be an electromagnetic solenoid type. In this case, the operation signal output from the operation device 26 may be directly input to the control valve unit 17 (that is, to the electromagnetic solenoid type direction change valve).

[0056] Note that the operation device 26 may be a hydraulic pilot type. Specifically, the operation device 26 uses the hydraulic oil supplied from the pilot pump 15 through the pilot line and outputs a pilot pressure corresponding to the operation content to the secondary side pilot line. And the secondary side pilot line is connected to the control valve unit 17. Thereby, a pilot pressure corresponding to the operation content regarding various driven elements (hydraulic actuators) in the operation device 26 can be input to the control valve unit 17. Therefore, the control valve unit 17 can drive each hydraulic actuator according to the operation content of the operation device 26 by an operator or the like. In this case, an operation sensor 29 capable of acquiring information regarding the operation state of the operation device 26 is provided, and the output of the operation sensor 29 is taken into the controller 30. Thereby, the controller 30 can grasp the operation state of the operation device 26. The operation sensor 29 is, for example, a pressure sensor that acquires information regarding the pilot pressure (operation pressure) of the secondary side pilot line of the operation device 26.

[0057] Also, part or all of the hydraulic actuators may be replaced with electric actuators. In this case, for example, the controller 30 may output an operation command corresponding to the operation content of the operation device 26 or the content of the remote operation defined by the remote operation signal to the electric actuator or a driver that drives the electric actuator. Further, when an operation signal is input from the operation device 26 to the electric actuator or a driver or the like, the electric actuator may be configured to be operable by the operation device 26.

[0058] In addition, when the excavator 100 is exclusively remotely operated or operates exclusively by a fully automatic driving function, the operating device 26 may be omitted.

[0059] The proportional valve 31 functions as a control valve for machine control and is provided for each driven element (hydraulic actuator) to be operated by the operating device 26 and for each operating direction (for example, the raising direction and the lowering direction of the boom 4) of the driven element (hydraulic actuator). For example, two proportional valves 31 are provided for each double-acting hydraulic actuator for driving the lower traveling body 1, the upper swing body 3, the boom 4, the arm 5, the bucket 6, and the like. The proportional valve 31 is provided, for example, in a pilot line between the pilot pump 15 and the control valve unit 17, and may be configured to be able to change its flow passage area (that is, the cross-sectional area through which the hydraulic oil can flow). Thereby, the proportional valve 31 can output a predetermined pilot pressure to the secondary-side pilot line by using the hydraulic oil of the pilot pump 15 supplied through the primary-side pilot line. Therefore, the proportional valve 31 can apply a predetermined pilot pressure corresponding to an operation command from the controller 30 to the control valve unit 17. Thus, for example, the controller 30 can directly supply a pilot pressure corresponding to the operation content (operation signal) of the operating device 26 from the proportional valve 31 to the control valve unit 17, and realize the operation of the excavator 100 based on the operation of the operator.

[0060] In addition, the controller 30 may control the proportional valve 31 to realize the automatic driving function of the excavator 100. Specifically, the controller 30 outputs an operation command corresponding to the automatic driving function to the proportional valve 31 from the proportional valve 31. Thereby, the controller 30 can realize the operation of the excavator 100 by the automatic driving function.

[0061] Further, the controller 30 controls the proportional valve 31 to realize the remote operation of the excavator 100. Specifically, the controller 30 outputs an operation command corresponding to the content of the operation specified by the operation signal received from the remote operation room RC to the proportional valve 31 by the communication device T1. Thereby, the controller 30 can supply a pilot pressure corresponding to the content of the remote operation from the proportional valve 31 to the control valve unit 17, and realize the operation of the excavator 100 based on the remote operation of the operator.

[0062] In addition, when the operating device 26 is a hydraulic pilot type, a shuttle valve may be provided in the pilot line between the operating device 26, the proportional valve 31, and the control valve unit 17. The shuttle valve has two inlet ports and one outlet port, and outputs the hydraulic oil having the higher pilot pressure among the pilot pressures input to the two inlet ports to the outlet port. The shuttle valve is provided for each driven element (hydraulic actuator) to be operated by the operating device 26 and for each operating direction of the driven element (hydraulic actuator), similar to the proportional valve 31. For example, two shuttle valves are provided for each double-acting hydraulic actuator for driving the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, the bucket 6, and the like. One of the two inlet ports of the shuttle valve is connected to the secondary pilot line of the operating device 26 (specifically, the above-mentioned lever device or pedal device included in the operating device 26), and the other is connected to the secondary pilot line of the proportional valve 31. The outlet port of the shuttle valve is connected to the pilot port of the corresponding direction switching valve of the control valve unit 17 through the pilot line. The corresponding direction switching valve is the direction switching valve that drives the hydraulic actuator that is the operation target of the above-mentioned lever device or pedal device connected to one inlet port of the shuttle valve. Therefore, each of these shuttle valves can apply the higher one of the pilot pressure of the secondary pilot line of the operating device 26 and the pilot pressure of the secondary pilot line of the proportional valve 31 to the pilot port of the corresponding direction switching valve. That is, the controller 30 can control the corresponding direction switching valve without depending on the operation of the operator on the operating device 26 by outputting a pilot pressure higher than the secondary pilot pressure of the operating device 26 from the proportional valve 31. Therefore, the controller 30 can control the operation of the driven elements (lower traveling body 1, upper slewing body 3, boom 4, arm 5, bucket 6) without depending on the operation state of the operator on the operating device 26, and can realize the automatic operation function and the remote operation function.

[0063] Also, when the operating device 26 is a hydraulic pilot type, in addition to the shuttle valve, a pressure reducing valve may be provided in the pilot line between the operating device 26 and the shuttle valve. The pressure reducing valve operates, for example, in response to a control signal input from the controller 30 and is configured to be able to change its flow passage area. Thereby, when the operating device 26 is being operated by the operator, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26. Further, the controller 30 can, for example, even when the operating device 26 is being operated, reduce the pilot pressure output from the operating device 26 with the pressure reducing valve to be lower than the pilot pressure output from the proportional valve 31. Therefore, by controlling the proportional valve 31 and the pressure reducing valve, the controller 30 can surely cause a desired pilot pressure to act on the pilot port of the direction switching valve in the control valve unit 17, for example, regardless of the operation content of the operating device 26. Thus, the controller 30 can more appropriately realize the automatic operation function and the remote operation function of the excavator 100, for example, by controlling the pressure reducing valve in addition to the proportional valve 31.

[0064] As shown in FIG. 3, the communication system of the excavator 100 according to the present embodiment includes a communication device T1.

[0065] The communication device T1 is connected to an external communication line and communicates with a device provided separately from the excavator 100. The device provided separately from the excavator 100 may include, in addition to the devices outside the excavator 100, portable terminal devices (mobile terminals) brought into the cab 10 by the user of the excavator 100. The communication device T1 may include, for example, a mobile communication module compliant with standards such as 4G (4th Generation) and 5G (5th Generation). Further, the communication device T1 may include, for example, a satellite communication module. Further, the communication device T1 may include, for example, a Wi-Fi communication module, a Bluetooth (registered trademark) communication module, or the like. Also, when there are a plurality of connectable communication lines, the communication device T1 may include a plurality of communication devices T1 according to the type of the communication line.

[0066] For example, the communication device T1 communicates with an external device such as a remote control room in the work site through a local communication line constructed in the work site. The local communication line is, for example, a mobile communication line by local 5G (so-called local 5G) constructed in the work site or a local network by Wi-Fi.

[0067] Further, the communication device T1 is configured to transmit and receive information to and from a communication device installed in a remote control room through a wide-area communication line including the work site, that is, a wide-area network.

[0068] In the present embodiment, a case will be described in which the engine 11 is used as a drive source and the hydraulic pump is operated by the driving force generated by the engine 11 to operate the attachment AT, swing the upper swing body 3, and travel. However, the present embodiment does not limit the drive source to the engine 11, and a motor may be used as the drive source. That is, the control described in the present embodiment may be applied to a so-called electric excavator in which the motor as the drive source is driven by the electric power supplied from the battery, or may be applied to an excavator equipped with a plurality of drive sources.

[0069] <Outline of processing performed by the controller> The controller 30 according to this embodiment can also perform operations with the fully automatic operation function by referring to the design data stored in the design data storage unit D4A (see FIG. 4) of the auxiliary storage device D4. In order for the controller 30 according to this embodiment to perform operations with the fully automatic operation function, it is necessary to grasp the three-dimensional shape of the ground around the excavator 100. For this reason, the controller 30 acquires three-dimensional point cloud data including the shapes and distances of objects existing around as measurement information of the space recognition device S7 mounted on the excavator 100. The ground around the excavator 100 according to this embodiment may be the ground at the work site of the excavator 100. For example, it includes the entire ground where construction is carried out (including slopes), the road surface on which it travels, and the slope surface after construction, etc.

[0070] As described above, in order for the controller 30 to autonomously control the excavator 100 with the fully automatic operation function, it is necessary to recognize the three-dimensional shape of the ground around the excavator 100. However, when measuring the surroundings of the excavator 100 with the space recognition device S7, there may be areas that are not measured by the space recognition device S7 due to the objects existing around the excavator 100, the terrain around the excavator 100, or the three-dimensional shape of the ground at the measurement target.

[0071] When areas that are not measured by the space recognition device S7 occur, it is difficult for the excavator 100 to perform operations on such areas and it is difficult to set such areas as the travel route when the excavator 100 performs operations with the fully automatic operation function.

[0072] In other words, in order for the excavator 100 to perform operations on areas that cannot be measured by the space recognition device S7 or to set a travel route including such areas, it is necessary to complement such unmeasurable areas.

[0073] Therefore, this embodiment prepares in advance a second learned model LM2 (see FIG. 4) obtained by machine learning the characteristics of the ground to be constructed.

[0074] Then, by using the second learned model LM2, the controller 30 according to the present embodiment receives three-dimensional shape data that has been complemented considering the characteristics of the ground for areas that have not been measured. By using the three-dimensional shape data, the controller 30 can perform operations on the ground around the excavator 100 regardless of whether there are areas that have not been measured.

[0075] <Block configuration of the excavator controller> FIG. 4 is a functional block diagram showing a configuration example of the controller 30 of the excavator 100 according to the present embodiment. In the example shown in FIG. 4, the block configuration of the controller 30 of the excavator 100 is shown.

[0076] The controller 30 receives information output by the boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, body tilt sensor S4, swing angle sensor S5, imaging device S6, space recognition device S7, input device D2, communication device T1, positioning device PS, etc. Then, based on the received information and the information stored in the auxiliary storage device D4, the controller 30 executes various calculations and outputs the calculation results to the first display device D3, the proportional valve 31, etc.

[0077] Note that, in the present embodiment, an example in which the controller 30 controls the excavator 100 will be described, but a part of the functions of the controller 30 may be realized by another controller (control device). That is, the functions of the controller 30 may be distributed and realized by a plurality of controllers mounted on the excavator 100.

[0078] For example, based on the inputs of the boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, body tilt sensor S4, and swing angle sensor S5, the controller 30 can grasp (estimate) the position of the tip (bucket 6) of the attachment AT. Therefore, while grasping the position of the tip of the attachment AT, the controller 30 can control the operation of the excavator 100 by its automatic driving function.

[0079] The excavator 100 drives operating elements (hereinafter referred to as "driven elements"), such as the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, and the bucket 6, by operating an actuator (for example, a hydraulic actuator) according to the operation of an operator boarding the cab 10.

[0080] Alternatively, or in addition to being configured to be operable by an operator in the cab 10, the excavator 100 may be configured to be remotely operated (remote-controlled) from outside the excavator 100. When the excavator 100 is remotely operated, the inside of the cab 10 may be unmanned.

[0081] Further, the excavator 100 may automatically operate the actuator regardless of the content of the operator's operation. Thereby, the excavator 100 realizes a function of automatically operating at least a part of the driven elements, such as the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, and the bucket 6, that is, a so-called "automatic operation function" or "machine control function".

[0082] The automatic driving function may include a function of automatically operating a driven element (actuator) other than the driven element (actuator) of the operation target in response to an operation on the operation device 26 of the operator or a remote operation, that is, a so-called "semiautomatic operation function" or an "operation support type machine control function". Further, the automatic driving function may include a function of automatically operating at least a part of a plurality of driven elements (hydraulic actuators) on the premise that there is no operation on the operation device 26 of the operator or a remote operation, that is, a so-called "fully automatic driving function" or a "fully automatic type machine control function". In the excavator 100, when the fully automatic driving function is valid, the inside of the cab 10 may be unmanned. Further, the semiautomatic operation function, the fully automatic operation function, etc. may include a mode in which the operation content of the driven element (actuator) of the automatic driving target is automatically determined according to a rule defined in advance. Further, the semiautomatic operation function, the fully automatic operation function, etc. may include a mode (so-called "automatic driving function") in which the excavator 100 autonomously makes various determinations, and the operation content of the driven element (hydraulic actuator) of the automatic driving target is determined autonomously according to the determination result.

[0083] Specifically, when the arm 5 is being operated by the operator through the operation device 26, the controller 30 may automatically operate at least one of the boom 4 and the bucket 6 so that a preset target design surface (hereinafter simply referred to as "design surface") coincides with the tip position of the bucket 6. Further, the controller 30 may also automatically operate the arm 5 regardless of the operation state of the operation device 26 that operates the arm 5. That is, the controller 30 may cause the attachment to perform a preset operation by using the operation of the operation device 26 by the operator as a trigger. Hereinafter, the function of the controller 30 that operates at least one of the boom 4 and the bucket 6 in addition to the arm 5 in response to the operation of the operation device 26 corresponding to the arm 5 is referred to as the "semiautomatic operation function". The semiautomatic operation function may be executed, for example, by operating a predetermined switch (hereinafter, "MC (Machine Control) switch") arranged at the tip of any one of the lever devices included in the operation device 26.

[0084] The auxiliary storage device D4 stores a design data storage unit D4A, a first learned model LM1, and a second learned model LM2.

[0085] The design data storage unit D4A stores design data. The design data includes construction data indicating the three-dimensional shape after the excavator 100 has performed construction at the work site. The construction data includes position data of the construction target in the world coordinate system indicated by GNSS and three-dimensional shape data after construction. For example, the design data includes position data and three-dimensional shape data of a target design surface formed after the excavator 100 has excavated earth and sand.

[0086] The position data is expressed in a reference coordinate system similar to the position data acquired by GNSS, for example. The reference coordinate system is, for example, the World Geodetic System. The World Geodetic System is a three-dimensional orthogonal XYZ coordinate system with the origin at the center of gravity of the earth, the X-axis in the direction of the intersection of the Greenwich meridian and the equator, the Y-axis in the direction of 90 degrees east longitude, and the Z-axis in the direction of the North Pole.

[0087] When the measurement information of the space recognition device S7 and the image information captured by the imaging device S6 are input from the input layer, the first learned model LM1 outputs from the output layer the measurement information obtained by removing the three-dimensional shape of the objects existing around the excavator 100 from the measurement information of the space recognition device S7 and the image information obtained by removing the area where the objects existing around the excavator 100 are captured from the image information captured by the imaging device S6.

[0088] A plurality of second learned models LM2 are provided according to the environment of the object to be constructed by the excavator 100. For example, the second learned models LM2 are provided for each environment such as a construction work site, a civil engineering work site, a mine, a demolition work site, forestry, an industrial waste treatment site, a metal recycling treatment site, and an agricultural work site.

[0089] When the measurement information showing the three-dimensional shape of the ground around the excavator 100 and the image information showing the ground around the excavator 100 are input from the input layer, the three-dimensional shape data obtained by coloring the three-dimensional shape of the ground around the excavator 100 is output as three-dimensional shape data in which the three-dimensional shape and color of the area not measured by the space recognition device S7 and the imaging device S6 are complemented.

[0090] As the machine learning used for generating the first learned model LM1 and the second learned model LM2, for example, a neural network may be applied. Specifically, it is machine learning using a deep neural network (DNN), and deep learning (deep neural network) may be applied. As deep learning, for example, a convolutional neural network, RNN (Recurrent Neural Networks), or LSTM (Long Short Term Memory) may be applied.

[0091] The first learned model LM1 and the second learned model LM2 are generated by performing machine learning based on a teacher dataset generated in advance in an information processing device (not shown).

[0092] Specifically, the first learned model LM1 is subjected to machine learning processing using the measurement information of the space recognition device S7 and the image information captured by the imaging device S6 included in the teacher dataset, and the three-dimensional shape data obtained by removing the three-dimensional shape of the objects existing around the excavator 100 from the measurement information of the space recognition device S7, and the image information obtained by removing the area where the objects existing around the excavator 100 are captured from the image information captured by the imaging device S6. Note that the machine learning is not limited to the above-described method, and may be performed as long as the objects existing around the excavator 100 can be removed.

[0093] For each environment where construction is to be carried out, a teacher dataset is prepared for the second learned model LM2. Then, each of the plurality of second learned models LM2 is subjected to machine learning processing using the teacher dataset generated for each environment where construction is to be carried out.

[0094] Specifically, each of the plurality of second learned models LM2 is subjected to machine learning processing using a teacher dataset that includes measurement information indicating the three-dimensional shape of the ground around the excavator 100, image information showing the ground around the excavator 100, and three-dimensional shape data with color applied to the ground around the excavator 100, where there are no areas that cannot be measured by the space recognition device S7 and the imaging device S6 (in other words, the three-dimensional shape and color are complemented).

[0095] Furthermore, the second learned model LM2 may be prepared for each soil characteristic at the work site. The second learned model LM2 can recognize the state of the soil at the work site, such as the size of the soil at the work site, the angle of repose of the soil at the work site, the type of soil at the work site, the firmness of the soil at the work site, and the appearance of the soil at the work site, through machine learning of the teacher data prepared for each soil characteristic. In other words, when complementing areas that cannot be measured, the second learned model LM2 can perform complementation considering the above-mentioned soil characteristics.

[0096] Note that the first learned model LM1 and the second learned model LM2 may be updated by adding and training new teacher datasets to the existing first learned model LM1 and second learned model LM2.

[0097] For example, before the excavator 100 performs work, measurement information measured by the spatial recognition device S7 and image information obtained by imaging the work site where the work will be performed with the imaging device S6 are prepared in advance for the work site where the work will be performed. Then, among the plurality of second learned models LM2, the second learned model LM2 corresponding to the work site is selected. Then, the selected second learned model LM2 is subjected to additional machine learning processing using teacher data generated based on the prepared measurement information and image information. The additional machine learning processing may be performed by the controller 30 or by another information processing device. In the present embodiment, by performing additional machine learning corresponding to the work site where the work will be performed, it is possible to complement the three-dimensional shape and color suitable for the work site, so that the accuracy of the complement can be improved.

[0098] The controller 30 includes an acquisition unit 301, a deletion unit 302, a complementation unit 303, a generation unit 304, and an automatic control unit 305.

[0099] The acquisition unit 301 acquires various information from various sensors. For example, the acquisition unit 301 acquires image information captured by the imaging device S6 (front camera S6F, left camera S6L, right camera S6R, and rear camera S6B). For example, the acquisition unit 301 acquires measurement information indicating the results measured by the spatial recognition device S7 (rear spatial recognition device S7B, left spatial recognition device S7L, right spatial recognition device S7R, and front spatial recognition device S7F).

[0100] The acquisition unit 301 acquires detection information detected by each of the boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, machine body tilt sensor S4, and slewing angle sensor S5. The acquisition unit 301 acquires the position and orientation of the excavator 100 from the positioning device PS.

[0101] The acquisition unit 301 acquires design data corresponding to the work site where the excavator 100 is working from the design data storage unit D4A.

[0102] The deletion unit 302 inputs the measurement information and the image information acquired by the acquisition unit 301 into the input layer of the first pre-trained model LM1, and receives from the output layer of the first pre-trained model LM1 the measurement information obtained by removing the three-dimensional shape of the object existing around the excavator 100 from the measurement information, and the image information obtained by removing the area where the object existing around the excavator 100 is shown from the image information. In the present embodiment, before the complementation unit 303 described later complements the three-dimensional shape of the ground, by deleting the information regarding the object existing around the excavator 100, it is possible to suppress the influence of the surrounding objects on the complementation during the complementation of the ground, and realize an improvement in the complementation accuracy. The object to be deleted may be any object other than the ground existing at the work site, and examples thereof include a dump truck, a work machine including an excavator, and a building such as a prefabricated house.

[0103] FIG. 5 is a diagram showing the three-dimensional space around the excavator 100 based on the measurement information and the image information acquired by the acquisition unit 301. In the example shown in FIG. 5, for ease of explanation, it represents a state in which the color of the image information is given to the three-dimensional shape shown by the measurement information.

[0104] In the example shown in FIG. 5, a region 1500 representing the three-dimensional shape of the object and the ground existing around the excavator 100 is shown with the region 1501 where the excavator 100 was present as the center.

[0105] The region 1500 includes the three-dimensional shape 1511 of another excavator, the three-dimensional shape 1512 of a dump truck, and the three-dimensional shape 1513 of an earth and sand mountain.

[0106] Furthermore, in the area 1500, there is ground that is not measured due to dead angles formed by the object, the surrounding terrain, or the three-dimensional shape of the measurement target itself. As areas that are not measured due to an object existing between the spatial recognition device S7 and the ground, for example, there are a first area 1521 and a second area 1522. The reasons why the first area 1521 and the second area 1522 are not measured will be explained. First, between the first area 1521 and the spatial recognition device S7, there is the three-dimensional shape 1511 of another shovel. The signal (such as laser light) transmitted from the spatial recognition device S7 is reflected by the three-dimensional shape 1511 of the other shovel, and the signal does not reach the first area 1521. For this reason, the first area 1521 becomes an area that is not measured. Similarly, between the second area 1522 and the spatial recognition device S7, there is the three-dimensional shape 1512 of a dump truck. The signal (such as laser light) transmitted from the spatial recognition device S7 is reflected by the three-dimensional shape 1512 of the dump truck, and the signal does not reach the second area 1522. For this reason, the second area 1522 becomes an area that is not measured. That is, the first area 1521 and the second area 1522 have a three-dimensional shape such that the object existing between them and the spatial recognition device S7 reflects the signal from the spatial recognition device S7, so they become areas that are not measured.

[0107] As areas that cannot be measured due to the terrain existing between the spatial recognition device S7 and the ground or the three-dimensional shape of the ground, for example, there are a third area 1523 and a fourth area 1524. The reasons why the third area 1523 and the fourth area 1524 are not measured will be explained.

[0108] First, there is a three-dimensional shape of the earth and sand mountain (an example of terrain) 1513 between the third area 1523 and the space recognition device S7. A signal (such as laser light) transmitted from the space recognition device S7 is reflected by the three-dimensional shape 1513 of the earth and sand mountain, and the signal does not reach the third area 1523. For this reason, the third area 1523 becomes an area that has not been measured. That is, since the terrain existing between the third area 1523 and the space recognition device S7 reflects the signal from the space recognition device S7, the third area 1523 becomes an area that cannot be measured. The terrain that reflects the signal from the space recognition device S7 is not limited to an earth and sand mountain, and may be, for example, a cliff or other terrain.

[0109] The fourth area 1524 is a hole provided in the ground (an example of the three-dimensional shape of the ground). Since a signal (such as laser light) transmitted from the space recognition device S7 does not reach the bottom of the hole existing in the fourth area 1524, the fourth area 1524 becomes an area that has not been measured. That is, the fourth area 1524 becomes an area that has not been measured due to the three-dimensional shape of the ground existing in the fourth area 1524. The three-dimensional shape of the ground that has not been measured is not limited to a hole, and may be, for example, under a cliff, on the opposite side of an earth and sand mountain, or other three-dimensional shapes of the ground.

[0110] By the way, when the controller 30 in the present embodiment executes the fully automatic driving function, it is necessary to grasp the three-dimensional shape of the ground of the excavator 100.

[0111] However, when the controller 30 refers to the measurement information and image information used in the three-dimensional space of FIG. 5, information is missing for the first area 1521, the second area 1522, the third area 1523, and the fourth area 1524. That is, it is difficult for the controller 30 to perform work considering the first area 1521, the second area 1522, the third area 1523, and the fourth area 1524.

[0112] Therefore, the controller 30 according to the present embodiment performs processing for complementing the ground of the first area 1521, the second area 1522, the third area 1523, and the fourth area 1524.

[0113] First, the deletion unit 302 deletes the three-dimensional shape of the objects existing around the excavator 100 from the measurement information and the image information acquired by the acquisition unit 301.

[0114] FIG. 6 is a diagram showing the three-dimensional space around the excavator 100 based on the measurement information and the image information after the deletion unit 302 deletes the information of the objects existing around the excavator 100. In the example shown in FIG. 6, for ease of explanation, it represents a state in which the color of the image information is assigned to the three-dimensional shape data indicated by the measurement information.

[0115] In the example shown in FIG. 6, a region 1600 representing the three-dimensional shape of the ground around the excavator 100 is shown with the region 1501 where the excavator 100 existed as the center.

[0116] The region 1600 has had the three-dimensional shape 1511 of another excavator and the three-dimensional shape 1512 of the dump truck deleted from the region 1500 in FIG. 5. The three-dimensional shape 1513 of the earth and sand mound remains because it is part of the three-dimensional shape of the ground.

[0117] In the region 1600, there are a first region 1521, a second region 1522, a third region 1523, and a fourth region 1524 that are blind spots due to the shape of the object or the ground and could not be measured. The first region 1521, the second region 1522, the third region 1523, and the fourth region 1524 will not be described further as they are the same as in FIG. 5.

[0118] The complementation unit 303 complements the three-dimensional shape of the ground that was not measured by the space recognition device S7 based on the measurement information measured by the space recognition device S7. In the complementation unit 303 according to the present embodiment, in addition to the measurement information, the color of the ground that was not imaged by the imaging device S6 is complemented using the image information imaged by the imaging device S6. The ground to be complemented is, for example, any one or more of earth and sand, roads, etc.

[0119] In this embodiment, an example of using the second pre-trained model LM2 to complement the three-dimensional shape and the color of the ground will be described, but the method of using the second pre-trained model LM2 is not limited. As a method for complementing the measurement information and the image information, well-known methods may be used. For example, a method of complementing the three-dimensional shape and the color of the unmeasured area using the three-dimensional shape and the color of the measured area may be used.

[0120] The complementing unit 303 selects the second pre-trained model LM2 corresponding to the work site from among a plurality of second pre-trained models LM2. Any method may be used for the selection method, and it may be selected by the operator or automatically selected based on information regarding the work site. In this embodiment, by using the second pre-trained model LM2 corresponding to the work site, it becomes possible to complement the three-dimensional shape of the ground suitable for the work site, so that an improvement in the accuracy of the complement can be realized.

[0121] Furthermore, the selected second pre-trained model LM2 is subjected to additional machine learning processing using teacher data generated based on measurement information and image information representing the work site to be worked on next. Thereby, it becomes possible to perform complementing suitable for the work site.

[0122] Then, the complementing unit 303 inputs, into the input layer of the second pre-trained model LM2, measurement information (an example of first three-dimensional shape information) obtained by removing the three-dimensional shape of an object existing around the excavator 100, and image information obtained by removing the area in which the object existing around the excavator 100 is shown, and receives, from the second pre-trained model LM2, three-dimensional shape data representing a three-dimensional shape with colors applied to the ground around the excavator 100, which is three-dimensional shape data (an example of second three-dimensional shape information) in which the three-dimensional shape and colors are complemented for areas not measured by the spatial recognition device S7 and the imaging device S6. In the present embodiment, since the three-dimensional shape is complemented using the second pre-trained model LM2, it is possible to complement the three-dimensional shape in consideration of the characteristics of the ground, so that an improvement in the complementing accuracy can be realized. Further, in the present embodiment, by complementing the colors in addition to the three-dimensional shape, the situation of the ground around the excavator 100 can be recognized more specifically, so that an improvement in the accuracy of the work on the ground can be realized.

[0123] The three-dimensional shape data output from the second pre-trained model LM2 is, for example, information representing the three-dimensional shape of the ground around the excavator 100 in the form of a point cloud or a mesh, and is data in which each point in the point cloud is given a color. Note that the present embodiment shows an example of the three-dimensional shape data, and any data that can recognize the three-dimensional shape of the ground around the excavator 100 may be used.

[0124] FIG. 7 is a diagram showing the three-dimensional space around the excavator 100, represented by three-dimensional shape data in which the three-dimensional shape and colors are complemented for areas not measured by the complementing unit 303.

[0125] In the example shown in FIG. 7, a region 1700 representing the three-dimensional shape of the ground around the excavator 100 is shown, centered on the region 1501 where the excavator 100 was present.

[0126] Region 1700 is supplemented with a first region 1521, a second region 1522, a third region 1523, and a fourth region 1524 from region 1600 in FIG. 6. In the example shown in FIG. 7, the supplemented regions are represented as supplementary regions 1701, 1702, 1703, and 1704.

[0127] For example, the supplementary region 1703 is supplemented based on the three-dimensional shape of the ground at the work site and the characteristics of the soil at the work site.

[0128] For example, the supplementary region 1701 is supplemented with the three-dimensional shape 1713 of the soil pile by the ground shapes 1711 and 1712 (around the first region 1521) included in the measurement information, the angle of repose generated by the soil at the work site, the type of soil at the work site, and the firmness of the soil at the work site.

[0129] For example, the supplementary region 1702 is supplemented with the color 1722 of the ground by the color 1721 of the ground included in the captured image and the appearance of the soil at the work site.

[0130] For example, the supplementary region 1704 is supplemented with the three-dimensional shape 1741 of the hole by the shape of the region not measured by the measurement information, the angle of repose generated by the soil at the work site, the type of soil at the work site, and the firmness of the soil at the work site.

[0131] The generation unit 304 further adds the three-dimensional shape and color of the objects existing around the excavator 100 to the three-dimensional shape data supplemented by the supplementation unit 303 to generate information on a virtual three-dimensional space showing the positional relationship between the objects and the ground around the excavator 100.

[0132] The virtual three-dimensional space is, for example, a three-dimensional space with the central position of the excavator 100 as the origin, in which the three-dimensional shapes of the ground and objects existing around the excavator 100 are shown, and the ground and objects existing around the excavator 100 are colored.

[0133] The addition of the three-dimensional shape and color of the objects existing around the excavator 100 is performed based on the information of the objects deleted from the measurement information and the image information by the deletion unit 302. Further, the three-dimensional shape of the current excavator 100 may be arranged in the virtual three-dimensional space. Therefore, in the present embodiment, a virtual three-dimensional space showing the shape and positional relationship of each of the excavator 100, the ground around the excavator 100, and the objects around the excavator 100 is generated.

[0134] Based on the design data stored in the design data storage unit D4A and the information of the virtual three-dimensional space generated by the generation unit 304, the automatic control unit 305 autonomously operates the excavator 100.

[0135] The automatic control unit 305 generates a trajectory for operating the attachment AT or the lower traveling body 1, and outputs a control command (current command) to the proportional valve 31 so that the attachment AT or the lower traveling body 1 operates according to the trajectory. The proportional valve 31 individually adjusts the pilot pressure acting on the control valve corresponding to each actuator according to the control command, thereby autonomously operating each actuator. Note that the method of generating the trajectory is omitted from the description as a well-known method may be used.

[0136] For example, the automatic control unit 305 operates the attachment AT with respect to the ground represented in the virtual three-dimensional space while avoiding the objects represented in the virtual three-dimensional space so as to form the target design surface shown in the design data.

[0137] Furthermore, the automatic control unit 305 generates a movement path so as to avoid the objects represented in the virtual three-dimensional space and travel on the ground represented in the virtual three-dimensional space in order to move to the destination shown in the design data. Then, the automatic control unit 305 operates the lower traveling body 1 so as to travel according to the movement path.

[0138] In this way, by using the information of the virtual three-dimensional space generated by the generation unit 304, the automatic control unit 305 can perform operations such as work or travel on the ground around the excavator 100 including the area where dead angles occur (unmeasurable areas) in the space recognition device S7 and the imaging device S6.

[0139] Even if the three-dimensional shape and color complementation are different from the actual ground, at least the automatic control unit 305 can set the work or travel route for the ground. For example, in excavation on the ground, the amount of earth and sand loaded in the bucket 6 may be different from the estimate. However, in terms of enabling work on the ground, it realizes an improvement in work efficiency. Also, it enables setting of the travel route for the complemented ground. That is, when traveling on the complemented ground, since the three-dimensional shape of the complemented ground is different from the actual one, it may be necessary to go up or down the ground, but at least in terms of enabling movement on the ground to reach the destination, it realizes an improvement in work efficiency.

[0140] Next, the processing procedure executed by the controller 30 according to this embodiment will be described. FIG. 8 is a flowchart showing the processing procedure until the controller 30 according to this embodiment performs automatic control.

[0141] First, the acquisition unit 301 acquires the image information captured by the imaging device S6 and the measurement information measured by the space recognition device S7 (S1801).

[0142] The deletion unit 302 inputs the measurement information and the image information to the input layer of the first pre-trained model LM1, and receives from the first pre-trained model LM1 the measurement information and the image information from which the information on the objects existing around the excavator 100 has been deleted (S1802).

[0143] The completion unit 303 selects a second pre-trained model LM2 corresponding to the work site from among a plurality of second pre-trained models LM2 (S1803). This specific method may be selected by an operation from the operator, or the completion unit 303 may automatically select the second pre-trained model LM2 corresponding to the work site based on the information of the work site.

[0144] When the measurement information and the image information, from which information about the objects existing around the excavator 100 has been deleted, are input from the input layer of the second pre-trained model LM2, the completion unit 303 receives three-dimensional shape data that represents a three-dimensional shape with color applied to the ground around the excavator 100 and that complements the three-dimensional shape and color for the area not measured by the spatial recognition device S7 and the imaging device S6 (S1804).

[0145] The generation unit 304 adds the three-dimensional data and color of the objects existing around the excavator 100 to the three-dimensional shape data complemented by the completion unit 303, and generates information on a virtual three-dimensional space showing the positional relationship between the objects and the ground around the excavator 100 (S1805).

[0146] The automatic control unit 305 performs automatic control of the excavator 100 based on the design data stored in the design data storage unit D4A and the information on the virtual three-dimensional space generated by the generation unit 304 (S1806).

[0147] In this embodiment, by having the controller 30 include the above-described configuration, the three-dimensional shape of the ground around the excavator 100 can be recognized, enabling automatic control that takes into account the three-dimensional shape of the ground.

[0148] In this embodiment, the case where the controller 30 performs the above-described processing has been described. However, this embodiment is not limited to the method in which the controller 30 performs the above-described processing. For example, the controller 30 may transmit the image information and the measurement information to an external server via the communication device T1. Then, the external server may perform the above-described processing, complement the three-dimensional shape and color of the ground around the excavator 100, and transmit information on a virtual three-dimensional space showing the positional relationship between the objects and the ground around the excavator 100 to the communication device T1. Then, the controller 30 may autonomously operate the excavator 100 based on the received virtual three-dimensional space information.

[0149] Also, in this embodiment, the method of autonomously controlling the excavator 100 based on the information of the virtual three-dimensional space has been described. However, this embodiment is not limited to the method of autonomously controlling the excavator 100. For example, when the semi-automatic operation function is executed by operating a predetermined switch by the operator, the controller 30 may operate the attachment AT or the lower traveling body 1 based on the information of the virtual three-dimensional space. As another example, the first display device D3 may display the information of the virtual three-dimensional space generated by the generation unit 304. Then, the operator may operate the excavator 100 after checking the surrounding situation with reference to the information displayed on the first display device D3.

[0150] (Modification example) The above-described embodiment has described an example of automatically controlling the excavator 100 based on the measurement information of the space recognition device S7 and the image information of the imaging device S6. However, the above-described embodiment is not limited to the method of using the measurement information of the space recognition device S7 and the image information of the imaging device S6. Therefore, as a modification example, the case of using the measurement information of the space recognition device S7 will be described. The first learned model LM1 and the second learned model LM2 according to this modification example only need to input the measurement information.

[0151] The deletion unit 302 according to this modification example receives measurement information obtained by deleting the three-dimensional shape of an object existing around the excavator 100 by inputting the measurement information of the space recognition device S7 into the first learned model LM1.

[0152] The complementation unit 303 according to this modification example receives three-dimensional shape data obtained by complementing the three-dimensional shape of the ground in an area not measured by the space recognition device S7 by inputting the measurement information into the second learned model LM2.

[0153] The generation unit 304 according to this modification example adds the three-dimensional shape of an object existing around the excavator 100 to the three-dimensional shape data complemented by the complementation unit 303, and generates information on a virtual three-dimensional space showing the positional relationship between the objects and the ground around the excavator 100.

[0154] The automatic control unit 305 according to this modification example autonomously operates the excavator 100 based on the design data stored in the design data storage unit D4A and the information on the virtual three-dimensional space generated by the generation unit 304.

[0155] The controller 30 according to this modification example can recognize the three-dimensional shape of the ground around the excavator 100 based on the measurement information measured by the space recognition device S7 without using the image information captured by the imaging device S6. Therefore, it has the same effects as the above-described embodiment and can achieve cost reduction.

[0156] (Second Embodiment) In the above-described embodiment and modification example, the case where the controller 30 of the excavator 100 recognizes the surrounding situation and performs automatic control has been described. However, the above-described embodiment and modification example do not limit the recognition of the surrounding situation for the purpose of performing automatic control. Therefore, in this embodiment, the case of monitoring the surroundings of the excavator 100 will be described.

[0157] FIG. 9 is a schematic diagram showing a configuration example of the support system SYS of the excavator 100 according to the present embodiment. FIG. 9 shows an example in which the excavator 100, the fixed-point measurement device 2150, the information center 2100, and the remote operation room RC are connected.

[0158] The excavator 100 uses the communication device T1 provided in the excavator 100 to transmit the detection results from various sensors provided in the excavator 100 to the information center 2100. For example, the excavator 100 transmits the image information captured by the imaging device S6 and the measurement information measured by the space recognition device S7 to the information center 2100.

[0159] Moreover, the above-described embodiment is not limited to the method of providing the imaging device S6 and the space recognition device S7 on the excavator 100. Therefore, in the present embodiment, an example in which the fixed-point measurement device 2150 is provided at the work site is assumed. As an example different from the present embodiment, when the fixed-point measurement device 2150 is provided at the work site, a mode in which the imaging device S6 and the space recognition device S7 are not mounted on the excavator 100 is also conceivable.

[0160] One or more fixed-point measurement devices 2150 are installed at the work site where the excavator 100 performs work. The fixed-point measurement device 2150 has functions as the imaging device S6 and the space recognition device S7 described in the above-described embodiment. For example, the fixed-point measurement device 2150 transmits the image information and the measurement information to the information center 2100.

[0161] The information center 2100 according to the present embodiment has the same configuration as the controller 30 of the above-described embodiment. Therefore, the information center 2100 receives the image information and the measurement information from one or more of the excavator 100 and the fixed-point measurement device 2150.

[0162] The information center 2100 generates information on a virtual three-dimensional space that complements the areas that have not been measured at the work site. The information on the virtual three-dimensional space includes the three-dimensional shape of the ground at the work site and the three-dimensional shape of the work machine (for example, an excavator) existing at the work site.

[0163] The information center 2100 may combine and process the image information and measurement information received from the excavator 100 and the image information and measurement information received from the fixed-point measurement device 2150. The information center 2100 may perform processing to complement an unmeasured area after arranging the three-dimensional shape and color based on each of the plurality of image information and measurement information in a virtual three-dimensional space.

[0164] Then, the information center 2100 displays the information of the virtual three-dimensional space on the display device 2110. Thereby, the administrator of the information center 2100 can recognize the current progress status from the three-dimensional shape of the ground at the work site. Since the administrator of the work site can grasp the current situation of the ground at the work site, the progress of the construction can be managed and the work can be advanced according to the plan.

[0165] Furthermore, the information center 2100 may transmit the information of the virtual three-dimensional space to the excavator 100.

[0166] In this case, the excavator 100 performs automatic control based on the received information of the virtual three-dimensional space and the design data.

[0167] Furthermore, the excavator 100 according to the present embodiment may be remotely operated. In the support system SYS according to the present embodiment, a remote operation room RC is provided. The remote operation room RC is provided with a display device DR, an operation device R26, an operation sensor R29, an operation seat DS, and a remote controller R30, and a communication device T2.

[0168] The remote controller R30 may receive the information of the virtual three-dimensional space via the communication device T2 and display the information of the virtual three-dimensional space on the display device DR.

[0169] Furthermore, the remote controller R30 may have the same functions as the controller 30 of the above-described embodiment. In this case, the remote controller R30 may receive image information and measurement information via the information center 2100, perform the same processing as the controller 30, generate information on the virtual three-dimensional space, and display the information on the virtual three-dimensional space on the display device DR. Thereby, the operator OP present in the operator's seat DS can confirm the situation around the excavator 100 even when present in the remote operation room RC.

[0170] The operator OP present in the operator's seat DS in the remote operation room RC confirms the surrounding situation and operates the operating device R26 as necessary. Then, the operation sensor R29 detects the operation content received by the operating device R26.

[0171] Then, the remote controller R30 generates a control signal for operating the excavator 100 based on the detected operation content. Then, the communication device T2 transmits the generated control signal to the excavator 100. By transmitting the control signal, the remote controller R30 can control the excavator 100.

[0172] In the present embodiment, the excavator 100 may perform automatic control as in the first embodiment. Then, when the operator OP present in the remote operation room RC monitors the current situation of the work site and determines that it is better to operate by himself / herself, the remote controller R30 may switch the control so that the excavator 100 is operated by the operating device R26 in response to the operation from the operator OP.

[0173] In the present embodiment, when a manager or the like manages the work site, the three-dimensional shape of the ground in the area that has not been measured is complemented, so that the situation of the work site can be recognized, improving visibility and making it easier to grasp the current situation.

[0174] <Function> In the above-described embodiments and modifications, the controller 30, the information center 2100, or the remote controller R30 performs the above-described control to complement the three-dimensional shape of the ground in the area that cannot be measured, thereby enabling recognition of the three-dimensional shape of the ground around the excavator 100. Then, since the excavator 100 can perform operations in consideration of the three-dimensional shape of the ground, it is possible to suppress the suspension of operations due to the inability to recognize the shape of the ground and realize an improvement in work efficiency.

[0175] As described above, the embodiments of the system for a working machine and the excavator according to the present invention have been described. However, the present invention is not limited to the above-described embodiments. Within the scope described in the claims, various changes, modifications, substitutions, additions, deletions, and combinations are possible. Naturally, they also belong to the technical scope of the present invention.

Explanation of Reference Numerals

[0176] 100 Excavator 1 Lower Traveling Body 2 Swing Mechanism 3 Upper Swing Structure 4 Boom 5 Arm 6 Bucket S1 Boom Angle Sensor S2 Arm Angle Sensor S3 Bucket Angle Sensor S4 Machine Body Tilt Sensor S5 Swing Angle Sensor S6 Imaging Device S7 Spatial Recognition Device PS Positioning Device T1 Communication Device D4 Auxiliary Storage Device D4A Design Data Storage Unit LM1 First Learned Model LM2 Second Learned Model 30 Controller 301 Acquisition Unit 302 Deletion Unit 303 Completion Unit 304 Generation Unit 305 Automatic Control Unit RC Remote Operation Room R30 Remote Controller T2 Communication Device 2100 Information Center 2110 Display Device 2150 Fixed Point Measuring Device

Claims

1. A space recognition device capable of measuring the three-dimensional shape of an object existing around a working machine, and a control device configured to complement the three-dimensional shape of the ground in an area not measured by the space recognition device based on measurement information including at least the three-dimensional shape of the ground around the working machine measured by the space recognition device, A system for a working machine comprising the above.

2. The ground not measured by the space recognition device is the ground in an area not measured due to an object existing between the space recognition device and the ground, or the ground in an area not measured due to the terrain existing between the space recognition device and the ground or the three-dimensional shape of the ground. The system for a working machine according to Claim 1.

3. Furthermore, a learned model subjected to machine learning processing using teacher data is provided, which outputs second three-dimensional shape information obtained by complementing the three-dimensional shape of an area not represented by the first three-dimensional shape information when the first three-dimensional shape information including the three-dimensional shape of the ground around the working machine is input, The control device inputs the first three-dimensional shape information including the three-dimensional shape of the ground around the working machine, which is included in the measurement information, into the learned model, and receives the second three-dimensional shape information obtained by complementing the three-dimensional shape of an area not represented by the first three-dimensional shape information. The system for a working machine according to Claim 1.

4. The learned model has been subjected to machine learning processing using the teacher data generated based on the result of measuring the three-dimensional shape of the ground on which construction is performed by the working machine. The system for a working machine according to Claim 3.

5. A plurality of the learned models are provided according to the environment in which the working machine performs construction, Each of the plurality of learned models has been subjected to machine learning processing using the teacher data generated for each environment in which construction is performed. The system for a working machine according to Claim 3.

6. Furthermore, an imaging device for imaging the periphery of the working machine is provided, When the learned model is further input with the image information captured by the imaging device, machine learning processing using teacher data is performed so as to output image information with the color of the area complemented. The control device inputs image information representing the color of the ground around the work machine into the learned model, and receives the second three-dimensional shape information to which a color is assigned based on the image information and the color of the area not shown in the image information is complemented. The system for a work machine according to claim 3.

7. The control device deletes the three-dimensional shape of an object existing around the work machine from the measurement information, and inputs the first three-dimensional shape information from which the three-dimensional shape of the object has been deleted into the learned model. The system for a work machine according to claim 3.

8. The space recognition device is attached to the work machine or installed on the ground where the work machine performs work. The system for a work machine according to claim 1.

9. Lower traveling body, An upper revolving body rotatably mounted on the lower traveling body, A space recognition device attached to the upper revolving body and capable of measuring the three-dimensional shape of an object existing around the excavator, A control device configured to complement the three-dimensional shape of the ground not measured by the space recognition device based on measurement information including at least the three-dimensional shape of the ground around the excavator measured by the space recognition device, An excavator comprising.

Images