System for work machine, and excavator

Abstract

To provide a system for a work machine capable of improving the accuracy of recognizing object-related situations.SOLUTION: A system for a work machine related to an embodiment of the disclosure includes: a spatial recognition unit that is capable of measuring the three-dimensional shape of a target that exists around the work machine; and a controller that is configured to complement the three-dimensional shape of the area not measured by the spatial recognition unit in the object on the basis of the three-dimensional shape included in the measurement information measured by the spatial recognition unit at least some of the area of the object.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a system for a working machine and an excavator.

Background Art

[0002] Conventionally, an object detection device is often provided to detect an object existing around a working machine (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Patent Document 1 describes calculating the coordinates of each object detected by an object detection device in a reference coordinate system to grasp the positional relationship between each object such as an obstacle and the excavator 100. However, when the object detection device detects an object, an area that is not measured in the object may occur depending on the positional relationship between the object detection device and the object. When an area that is not measured in the object occurs, it is difficult to perform control considering the object or monitor the periphery including the object.

[0005] One aspect of the present invention improves the recognition accuracy of the situation regarding an object by complementing an area that is not measured by a space recognition device among the three-dimensional shapes of the object.

Means for Solving the Problems

[0006] A system for a working machine according to one aspect of the present invention includes a spatial recognition device capable of measuring the three-dimensional shape of an object existing around the working machine, and a control device configured to complement the three-dimensional shape of a region of at least a part of the object included in measurement information measured by the spatial recognition device, for the three-dimensional shape of a region of the object that has not been measured by the spatial recognition device.

Advantages of the Invention

[0007] According to one aspect of the present invention, by complementing a region of the three-dimensional shape of an object that has not been measured by the spatial recognition device, the recognition accuracy of the situation regarding the object is improved.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, the embodiments described below are illustrative rather than limiting the invention, and not all features and combinations thereof described in the embodiments are necessarily essential to the invention. In each drawing, the same or corresponding components are denoted by the same or corresponding reference numerals, and the description thereof may be omitted.

[0010] Hereinafter, in the embodiments of the present invention, an example in which an excavator is used as an example of a working machine will be described, but the present invention is not limited to excavators. It may be applied to construction machines, standard machines, application machines, forestry machines, or transport machines based on hydraulic excavators.

[0011] (First Embodiment) First, with reference to FIG. 1, an overview of the excavator 100 according to the present embodiment will be described. FIG. 1 is a side view of the excavator 100 according to the first embodiment. FIG. 2 is a top view of the excavator 100 according to the first embodiment.

[0012] An upper swing body 3 is swingably mounted on a lower traveling body 1 of the excavator 100 via a swing mechanism 2. A boom 4 is attached to the upper swing body 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 as an end attachment is attached to the tip of the arm 5. The end attachment may be a slope bucket, a dredging bucket, or the like.

[0013] The boom 4, arm 5, and bucket 6 constitute an excavation attachment which is an example of the attachment AT, and are respectively hydraulically driven by the boom cylinder 7, arm cylinder 8, and bucket cylinder 9. A boom angle sensor S1 is attached to the boom 4, an arm angle sensor S2 is attached to the arm 5, and a bucket angle sensor S3 is attached to the bucket 6. The excavation attachment may be provided with a bucket tilt mechanism.

[0014] The boom angle sensor S1 detects the rotation angle of the boom 4. In the present embodiment, the boom angle sensor S1 is an acceleration sensor and can detect the boom angle which is the rotation angle of the boom 4 with respect to the upper swing body 3. The boom angle becomes the minimum angle, for example, when the boom 4 is lowered most, and increases as the boom 4 is raised.

[0015] The boom angle sensor S1 may include, for example, a rotary encoder, an acceleration sensor, a six-axis sensor, an IMU (Inertial Measurement Unit), etc. Further, the boom angle sensor S1 may also include a potentiometer using a variable resistor, a cylinder stroke sensor that detects the stroke amount of a hydraulic cylinder (boom cylinder 7) corresponding to the boom angle, etc. The same applies to the arm angle sensor S2, bucket angle sensor S3, and body inclination sensor S4 below. The detection signal corresponding to the boom angle by the boom angle sensor S1 is taken into the controller 30.

[0016] The arm angle sensor S2 detects the rotation angle of the arm 5. In the present embodiment, the arm angle sensor S2 is an acceleration sensor and can detect the arm angle which is the rotation angle of the arm 5 with respect to the boom 4. The arm angle becomes the minimum angle, for example, when the arm 5 is closed most, and increases as the arm 5 is opened.

[0017] The bucket angle sensor S3 detects the rotation angle of the bucket 6. In the present embodiment, the bucket angle sensor S3 is an acceleration sensor and can detect the bucket angle, which is the rotation angle of the bucket 6 with respect to the arm 5. The bucket angle, for example, has a minimum angle when the bucket 6 is closed most, and increases as the bucket 6 is opened.

[0018] The boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 may be a potentiometer using a variable resistor, a stroke sensor that detects the stroke amount of the corresponding hydraulic cylinder, or a rotary encoder that detects the rotation angle around the connecting pin, etc. The boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 constitute an attitude sensor that detects the attitude of the excavation attachment.

[0019] The upper slewing body 3 is provided with a cabin 10 which is a driver's cab and is equipped with a power source such as an engine 11. Further, a body tilt sensor S4, a slewing angle sensor S5, an imaging device S6, and a space recognition device S7 are attached to the upper slewing body 3. Also, a communication device T1 and a positioning device PS are attached to the upper slewing body 3.

[0020] The body tilt sensor S4 is configured to detect the tilt of the upper slewing body 3 with respect to a predetermined plane. In the present embodiment, the body tilt sensor S4 is an acceleration sensor that detects the tilt angle around the front-rear axis and the tilt angle around the left-right axis of the upper slewing body 3 with respect to the horizontal plane. The front-rear axis and the left-right axis of the upper slewing body 3, for example, are orthogonal to each other and pass through the excavator center point which is a point on the slewing axis of the excavator 100.

[0021] The slewing angle sensor S5 is configured to detect the slewing angular velocity of the upper slewing body 3. In the present embodiment, the slewing angle sensor S5 is a gyro sensor. The slewing angle sensor S5 may be a resolver or a rotary encoder, etc. The slewing angle sensor S5 may detect the slewing speed. The slewing speed may be calculated from the slewing angular velocity.

[0022] In addition, when the body tilt sensor S4 includes a gyro sensor, a 6-axis sensor, an IMU, etc. that can detect the angular velocity around three axes, the turning state (for example, the turning angular velocity) of the upper slewing body 3 may be detected based on the detection signal of the body tilt sensor S4. In this case, the turning angle sensor S5 may be omitted.

[0023] The imaging device S6 is configured to acquire images of the surroundings of the excavator 100. In the present embodiment, the imaging device S6 includes a front camera S6F that images the space in front of the excavator 100, a left camera S6L that images the space to the left of the excavator 100, a right camera S6R that images the space to the right of the excavator 100, and a rear camera S6B that images the space behind the excavator 100.

[0024] The imaging device S6 is, for example, a monocular camera having an imaging element such as a CCD or a CMOS, and outputs the captured image to the first display device D3 via the controller 30.

[0025] The input device D2 receives an operation input from the operator and outputs it to the controller 30. The input device D2 includes, for example, any hardware operation means such as a touch panel, a touch pad, buttons, toggles, and rotary knobs. Further, the input device D2 may include software operation means that can be operated through hardware operation means such as virtual button icons on an operation screen displayed on the first display device D3 or the like.

[0026] As shown in FIG. 2, the front camera S6F is attached to, for example, the roof of the cab 10. The left camera S6L is attached to the left end of the upper surface of the upper slewing body 3. The right camera S6R is attached to the right end of the upper surface of the upper slewing body 3. The rear camera S6B is attached to the rear end of the upper surface of the upper slewing body 3.

[0027] Note that the two-dot chain line in Fig. 2 represents the viewing angle (imaging range) of each imaging device S6 from the top view. Specifically, the imaging range CZ includes the imaging range CZB of the rear camera S6B, the imaging range CZF of the front camera S6F, the imaging range CZL of the left camera S6L, and the imaging range CZR of the right camera S6R. The four monocular cameras are preferably attached to the upper revolving body 3 so as not to protrude from the upper surface of the upper revolving body 3 as shown in Fig. 2.

[0028] In this embodiment, by providing the imaging device S6 in the above-described arrangement, an object existing around the excavator 100 can be imaged.

[0029] The space recognition device S7 is configured to recognize the state of the space around the excavator 100. The space recognition device S7 includes a rear space recognition device S7B that detects the space behind the excavator 100, a left space recognition device S7L that detects the space to the left of the excavator 100, a right space recognition device S7R that detects the space to the right of the excavator 100, and a front space recognition device S7F that detects the space in front of the excavator 100.

[0030] As an example, the space recognition device S7 uses LIDAR to measure the three-dimensional shape of an object existing around the excavator 100. LIDAR measures, for example, the distance between more than one million points within the monitoring range and the LIDAR. Note that this embodiment is not limited to the method using LIDAR, and any space recognition device capable of measuring the distance to an object may be used. For example, a stereo camera may be used, or a distance image camera or a ranging device such as a millimeter-wave radar may be used. When a millimeter-wave radar or the like is used as the space recognition device S7, a large number of signals (such as laser light) may be transmitted from the space recognition device S7 toward the object, and the distance and direction of the object may be derived from the reflected signal by receiving the reflected signal.

[0031] The rear space recognition device S7B is attached to the rear end of the upper revolving body 3. The left space recognition device S7L is attached to the left end of the upper revolving body 3. The right space recognition device S7R is attached to the right end of the upper revolving body 3. The front space recognition device S7F is attached to the front end of the upper surface of the cab 10.

[0032] Note that the dotted lines in Fig. 2 represent the viewing angles (monitoring ranges) of the upper surface views of the respective space recognition devices S7. Specifically, it includes the imaging range OZB of the rear space recognition device S7B, the imaging range OZF of the front space recognition device S7F, the imaging range OZL of the left space recognition device S7L, and the imaging range OZR of the right space recognition device S7R. The four monocular cameras are preferably attached to the upper revolving body 3 so as not to protrude from the upper surface of the upper revolving body 3 as shown in Fig. 2.

[0033] The space recognition device S7 may be configured to detect a predetermined object within a predetermined area set around the excavator 100. For example, the space recognition device S7 may have a human detection function configured to detect a human while distinguishing between a human and an object other than a human.

[0034] The communication device T1 is a device that controls communication between the excavator 100 and the outside. The communication device T1 controls, for example, wireless communication between an external GNSS (Global Navigation Satellite System) survey system and the excavator 100. The excavator 100 can acquire design data via wireless communication by using the communication device T1. However, the excavator 100 may acquire design data by using a semiconductor memory or the like. Note that the design data includes three-dimensional design data.

[0035] The positioning device PS is configured to acquire information regarding the position of the excavator 100 in the reference coordinate system. In the present embodiment, the positioning device PS is configured to measure the position and orientation of the excavator 100. Specifically, the positioning device PS is a GNSS receiver incorporating an electronic compass, and as the reference coordinate system, it measures the latitude, longitude, and altitude of the current position of the excavator 100, and also measures the orientation of the excavator 100.

[0036] FIG. 3 is a diagram showing a configuration example of the drive control system of the excavator 100 in FIG. 1. In FIG. 3, the mechanical power transmission system is shown by a double line, the hydraulic oil line is shown by a thick solid line, the pilot line is shown by a broken line, and the electric drive / control system is shown by a thin solid line, respectively.

[0037] The engine 11 is a power source of the excavator 100. In the present embodiment, the engine 11 is a diesel engine that employs isochronous control to maintain a constant engine speed regardless of the increase or decrease of the engine load. The fuel injection amount, fuel injection timing, boost pressure, etc. in the engine 11 are controlled by an engine control unit (ECU) D7.

[0038] The rotating shafts of the main pump 14 and the pilot pump 15, each serving as a hydraulic pump, are connected to the rotating shaft of the engine 11. The main pump 14 is connected to a control valve unit 17 via a hydraulic oil line.

[0039] The control valve unit 17 is a hydraulic control device that controls the hydraulic system of the excavator 100. Hydraulic actuators such as the left and right travel hydraulic motors, the boom cylinder 7, the arm cylinder 8, the bucket cylinder 9, and the swing hydraulic motor are connected to the control valve unit 17 via hydraulic oil lines. The swing hydraulic motor may be a swing electric generator.

[0040] FIG. 3 shows the connection relationship between the controller 30, the first display device D3, and the second display device D3S. In the present embodiment, the first display device D3 and the second display device D3S are connected to the controller 30. The first display device D3, the second display device D3S, and the controller 30 may be connected via a communication network such as CAN.

[0041] The space recognition device S7 can measure the three-dimensional shape of an object around the excavator 100, and can detect the presence or absence of an object existing in the space around the excavator 100, the distance to the object, and the like. The space recognition device S7 outputs the result of measuring the space to the controller 30 as measurement information.

[0042] The first display device D3 includes a conversion processing unit D3a that generates an image. In the present embodiment, the conversion processing unit D3a generates a camera image for display based on the output of a camera as the imaging device S6. The imaging device S6 is connected to the first display device D3 via, for example, a dedicated line.

[0043] The conversion processing unit D3a generates an image for display based on the output of the controller 30. In the present embodiment, the conversion processing unit D3a converts various information output by the controller 30 into an image signal. The information output by the controller 30 includes, for example, data indicating the temperature of the engine cooling water, data indicating the temperature of the hydraulic oil, data indicating the remaining fuel amount, data indicating the remaining amount of urea water, data indicating the position of the working part of the bucket 6, data indicating the orientation of the working surface of the work target, data indicating the orientation of the excavator 100, data indicating the operation direction for making the excavator 100 face the working surface directly, and the like.

[0044] Similar to the first display device D3, the second display device D3S includes a conversion processing unit D3Sa that generates an image. In the present embodiment, the second display device D3S is not directly connected to the imaging device S6. Therefore, the conversion processing unit D3Sa does not generate a camera image. However, the conversion processing unit D3Sa may generate a camera image when the second display device D3S is directly connected to the imaging device S6.

[0045] The conversion processing unit D3Sa generates a display image based on the output of the controller 30. In the present embodiment, the conversion processing unit D3Sa converts various types of information output by the controller 30 into an image signal.

[0046] The conversion processing unit D3a may be implemented as a function of the controller 30 instead of a function of the first display device D3. The same applies to the conversion processing unit D3Sa. In this case, the imaging device S6 is connected to the controller 30 instead of the first display device D3.

[0047] The first display device D3 and the second display device D3S operate by receiving power supply from the storage battery 70. The storage battery 70 is charged with the power generated by the alternator 11a (generator) of the engine 11. The power of the storage battery 70 is supplied to the electrical components 72 of the excavator 100 in addition to the controller 30, the first display device D3, and the second display device D3S. The starter 11b of the engine 11 is driven by the power from the storage battery 70 to start the engine 11.

[0048] The engine 11 is controlled by the engine controller unit D7. Various types of data indicating the state of the engine 11 are constantly transmitted from the engine controller unit D7 to the controller 30. The various types of data indicating the state of the engine 11 are an example of the operation information of the excavator 100, and include, for example, data indicating the cooling water temperature detected by the water temperature sensor 11c as an operation information acquisition unit. The controller 30 stores this data in a temporary storage unit (memory) 30a and can transmit it to the first display device D3 when necessary.

[0049] The following various types of data are supplied to the controller 30 as the operation information of the excavator 100 and stored in the temporary storage unit 30a of the controller 30.

[0050] For example, data indicating the swash plate tilt angle is supplied from a regulator 13 of a main pump 14, which is a variable displacement hydraulic pump, to a controller 30. Further, data indicating the discharge pressure of the main pump 14 is supplied from a discharge pressure sensor 14b to the controller 30. These data are stored in a temporary storage unit 30a. Further, an oil temperature sensor 14c is provided in a pipeline between a tank storing the hydraulic oil sucked by the main pump 14 and the main pump 14, and data representing the temperature of the hydraulic oil flowing through the pipeline is supplied from the oil temperature sensor 14c to the controller 30. The regulator 13, the discharge pressure sensor 14b, and the oil temperature sensor 14c are examples of an operation information acquisition unit.

[0051] Data indicating the fuel storage amount is supplied from a fuel storage amount detection unit 55a in a fuel storage unit 55 to the controller 30. In the present embodiment, data indicating the remaining fuel amount state is supplied from a fuel remaining amount sensor as the fuel storage amount detection unit 55a in a fuel tank as the fuel storage unit 55 to the controller 30.

[0052] Specifically, the fuel remaining amount sensor includes a float that follows the liquid level and a variable resistor (potentiometer) that converts the vertical movement amount of the float into a resistance value. With this configuration, the fuel remaining amount sensor can continuously display the remaining fuel amount state on a first display device D3. The detection method of the fuel storage amount detection unit can be appropriately selected according to the use environment or the like, and a detection method capable of stepwise displaying the remaining fuel amount state may be adopted. These configurations are the same for the urea water tank.

[0053] An operation device 26 is provided near the driver's seat in a cabin 10 and is used for an operator to operate various driven elements. Specifically, the operation device 26 is used for an operator to operate hydraulic actuators such as left and right traveling hydraulic motors, a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, and a slewing hydraulic motor. As a result, the operation of the driven elements driven by the hydraulic actuators by the operator can be realized. The operation device 26 includes a pedal device and a lever device for operating each driven element.

[0054] The operation sensor 29 is configured to detect the operation content of the operator using the operation device 26. In the present embodiment, the operation sensor 29 detects the operation direction and operation amount of the operation device 26 corresponding to each of the hydraulic actuators, and outputs an electrical signal (hereinafter also referred to as an operation signal) corresponding to the detected value to the controller 30. In the present embodiment, the controller 30 controls the opening area of the proportional valve 31 according to the output of the operation sensor 29. Then, the controller 30 supplies the hydraulic oil discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17. The pressure (pilot pressure) of the hydraulic oil supplied to each of the pilot ports is, in principle, a pressure corresponding to the operation direction and operation amount of the operation device 26 corresponding to each of the hydraulic actuators. Thus, the operation device 26 is configured to be able to supply the hydraulic oil discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17. Thereby, the hydraulic actuator can be driven.

[0055] Also, the direction switching valve for driving each hydraulic actuator built in the control valve unit 17 may be of an electromagnetic solenoid type. In this case, the operation signal output from the operation device 26 may be directly input to the control valve unit 17 (that is, to the electromagnetic solenoid type direction switching valve).

[0056] Note that the operating device 26 may be a hydraulic pilot type. Specifically, the operating device 26 utilizes the hydraulic oil supplied from the pilot pump 15 through the pilot line and outputs a pilot pressure corresponding to the operation content to the secondary pilot line. And the secondary pilot line is connected to the control valve unit 17. Thereby, a pilot pressure corresponding to the operation content regarding various driven elements (hydraulic actuators) in the operating device 26 can be input to the control valve unit 17. Therefore, the control valve unit 17 can drive each hydraulic actuator according to the operation content of the operating device 26 by the operator or the like. In this case, an operation sensor 29 capable of acquiring information regarding the operation state of the operating device 26 is provided, and the output of the operation sensor 29 is taken into the controller 30. Thereby, the controller 30 can grasp the operation state of the operating device 26. The operation sensor 29 is, for example, a pressure sensor that acquires information regarding the pilot pressure (operation pressure) of the secondary pilot line of the operating device 26.

[0057] Also, part or all of the hydraulic actuators may be replaced with electric actuators. In this case, for example, the controller 30 may output an operation command corresponding to the operation content of the operating device 26 or the content of the remote operation defined by the remote operation signal to the electric actuator or a driver or the like that drives the electric actuator. Further, when an operation signal is input from the operating device 26 to the electric actuator or a driver or the like, the electric actuator may be configured to be operable by the operating device 26.

[0058] Also, when the excavator 100 is solely remotely operated or solely operates by the fully automatic operation function, the operating device 26 may be omitted.

[0059] The proportional valve 31 functions as a control valve for machine control and is provided for each driven element (hydraulic actuator) to be operated by the operating device 26 and for each operating direction of the driven element (hydraulic actuator) (for example, the raising direction and the lowering direction of the boom 4). For example, two proportional valves 31 are provided for each double-acting hydraulic actuator for driving the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, the bucket 6, etc. The proportional valve 31 is provided, for example, in a pilot line between the pilot pump 15 and the control valve unit 17, and may be configured to be able to change its flow passage area (that is, the cross-sectional area through which the hydraulic oil can flow). Thereby, the proportional valve 31 can output a predetermined pilot pressure to the secondary-side pilot line by using the hydraulic oil of the pilot pump 15 supplied through the primary-side pilot line. Therefore, the proportional valve 31 can apply a predetermined pilot pressure according to an operation command from the controller 30 to the control valve unit 17. Thus, for example, the controller 30 can directly supply a pilot pressure corresponding to the operation content (operation signal) of the operating device 26 from the proportional valve 31 to the control valve unit 17, and realize the operation of the excavator 100 based on the operation of the operator.

[0060] Further, the controller 30 may control the proportional valve 31 to realize the automatic operation function of the excavator 100. Specifically, the controller 30 outputs an operation command corresponding to the automatic operation function to the proportional valve 31 from the proportional valve 31. Thereby, the controller 30 can realize the operation of the excavator 100 by the automatic operation function.

[0061] Further, the controller 30 controls the proportional valve 31 to realize the remote operation of the excavator 100. Specifically, the controller 30 outputs an operation command corresponding to the content of the operation specified by the operation signal received from the remote operation room RC to the proportional valve 31 by the communication device T1. Thereby, the controller 30 can supply a pilot pressure corresponding to the content of the remote operation from the proportional valve 31 to the control valve unit 17, and realize the operation of the excavator 100 based on the remote operation of the operator.

[0062] In addition, when the operating device 26 is a hydraulic pilot type, a shuttle valve may be provided in the pilot line between the operating device 26 and the proportional valve 31 and the control valve unit 17. The shuttle valve has two inlet ports and one outlet port, and outputs the hydraulic oil having the higher pilot pressure among the pilot pressures input to the two inlet ports to the outlet port. Similar to the proportional valve 31, the shuttle valve is provided for each driven element (hydraulic actuator) to be operated by the operating device 26 and for each operating direction of the driven element (hydraulic actuator). For example, two shuttle valves are provided for each double-acting hydraulic actuator for driving the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, the bucket 6, and the like. One of the two inlet ports of the shuttle valve is connected to the secondary pilot line of the operating device 26 (specifically, the above-described lever device or pedal device included in the operating device 26), and the other is connected to the secondary pilot line of the proportional valve 31. The outlet port of the shuttle valve is connected to the pilot port of the corresponding direction switching valve of the control valve unit 17 through the pilot line. The corresponding direction switching valve is the direction switching valve that drives the hydraulic actuator that is the operation target of the above-described lever device or pedal device connected to one inlet port of the shuttle valve. Therefore, each of these shuttle valves can apply the higher one of the pilot pressure of the secondary pilot line of the operating device 26 and the pilot pressure of the secondary pilot line of the proportional valve 31 to the pilot port of the corresponding direction switching valve. That is, the controller 30 can control the corresponding direction switching valve without depending on the operation of the operator on the operating device 26 by outputting a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26 from the proportional valve 31. Therefore, the controller 30 can control the operation of the driven elements (lower traveling body 1, upper slewing body 3, boom 4, arm 5, bucket 6) regardless of the operation state of the operator on the operating device 26, and can realize the automatic operation function and the remote operation function.

[0063] Also, when the operating device 26 is a hydraulic pilot type, in addition to the shuttle valve, a pressure reducing valve may be provided in the pilot line between the operating device 26 and the shuttle valve. The pressure reducing valve operates, for example, in response to a control signal input from the controller 30 and is configured to be able to change its flow passage area. Thereby, when the operating device 26 is operated by the operator, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26. Further, the controller 30 can reduce the pilot pressure output from the operating device 26 with the pressure reducing valve and make it lower than the pilot pressure output from the proportional valve 31, even when the operating device 26 is being operated, for example. Therefore, by controlling the proportional valve 31 and the pressure reducing valve, the controller 30 can surely cause a desired pilot pressure to act on the pilot port of the direction switching valve in the control valve unit 17, regardless of the operation content of the operating device 26, for example. Thus, the controller 30 can more appropriately realize the automatic operation function and the remote operation function of the excavator 100 by controlling the pressure reducing valve in addition to the proportional valve 31, for example.

[0064] As shown in FIG. 3, the communication system of the excavator 100 according to the present embodiment includes a communication device T1.

[0065] The communication device T1 is connected to an external communication line and communicates with a device provided separately from the excavator 100. The device provided separately from the excavator 100 may include, in addition to the devices outside the excavator 100, a portable terminal device (mobile terminal) brought into the cab 10 by the user of the excavator 100. The communication device T1 may include, for example, a mobile communication module compliant with standards such as 4G (4th Generation) and 5G (5th Generation). Further, the communication device T1 may include, for example, a satellite communication module. Further, the communication device T1 may include, for example, a Wi-Fi communication module, a Bluetooth (registered trademark) communication module, or the like. Also, when there are a plurality of connectable communication lines, the communication device T1 may include a plurality of communication devices T1 according to the type of the communication line.

[0066] For example, the communication device T1 communicates with an external device such as a remote control room in the work site through a local communication line constructed in the work site. The local communication line is, for example, a mobile communication line based on local 5G (so-called local 5G) constructed in the work site or a local network based on Wi-Fi.

[0067] Further, the communication device T1 is configured to transmit and receive information to and from a communication device installed in a remote control room through a wide-area communication line including the work site, that is, a wide-area network.

[0068] In the present embodiment, a case will be described in which the engine 11 is used as a drive source and the hydraulic pump is operated by the driving force generated by the engine 11 to perform the operation of the attachment AT, the turning operation of the upper swing body 3, and the traveling. However, the present embodiment does not limit the drive source to the engine 11, and a motor may be used as the drive source. That is, the control described in the present embodiment may be applied to a so-called electric excavator in which the motor as the drive source is driven by the electric power supplied from the battery, or may be applied to an excavator equipped with a plurality of drive sources.

[0069] <Outline of processing performed by the controller> The controller 30 according to this embodiment can also perform operations with a fully automatic driving function by referring to the design data stored in the design data storage unit D4A (see FIG. 4) of the auxiliary storage device D4. In order for the controller 30 according to this embodiment to perform operations with the fully automatic driving function, it is necessary to grasp the three-dimensional shape around the excavator 100. For this reason, the controller 30 acquires three-dimensional point cloud data including the three-dimensional shape and distance of the objects existing around as measurement information of the space recognition device S7 mounted on the excavator 100.

[0070] In order for the controller 30 to autonomously control the excavator 100 with the fully automatic driving function, it is necessary to recognize the three-dimensional shape of the objects existing around the excavator 100. However, when measuring around the excavator 100 with the space recognition device S7, depending on the positional relationship between the space recognition device S7 and the object, an area that cannot be measured in the object is generated.

[0071] That is, since there is an area where the space recognition device S7 cannot recognize the three-dimensional shape of the objects existing around the excavator 100, it may be difficult to operate the excavator 100 considering the object. For example, since the three-dimensional shape of the object on the side opposite to the space recognition device S7, in other words, the excavator 100, is unknown, it may be difficult to generate a movement path to avoid the object.

[0072] Therefore, this embodiment prepares in advance a second learned model LM2 obtained by machine learning the three-dimensional shape of the object.

[0073] Then, the controller 30 according to this embodiment receives the three-dimensional shape of the object in which the unmeasured area is complemented by using the second learned model LM2. By referring to the three-dimensional shape data of the object, the controller 30 can perform operations considering the three-dimensional shape of the objects existing around the excavator 100.

[0074] <Block Configuration of Excavator Controller> Figure 4 is a functional block diagram showing a configuration example of the controller 30 of the excavator 100 according to the present embodiment. In the example shown in FIG. 4, the block configuration of the controller 30 of the excavator 100 is shown.

[0075] The controller 30 receives information output from a boom angle sensor S1, an arm angle sensor S2, a bucket angle sensor S3, a machine body inclination sensor S4, a swing angle sensor S5, an imaging device S6, a space recognition device S7, an input device D2, a communication device T1, a positioning device PS, etc. Then, based on the received information and the information stored in the auxiliary storage device D4, various calculations are executed, and the calculation results are output to the first display device D3, the proportional valve 31, etc.

[0076] In the present embodiment, an example in which the controller 30 controls the excavator 100 will be described. However, a part of the functions of the controller 30 may be realized by another controller (control device). That is, the functions of the controller 30 may be realized in a distributed manner by a plurality of controllers mounted on the excavator 100.

[0077] For example, based on the inputs of the boom angle sensor S1, the arm angle sensor S2, the bucket angle sensor S3, the machine body inclination sensor S4, and the swing angle sensor S5, the controller 30 can grasp (estimate) the position of the tip of the attachment AT (bucket 6). Therefore, the controller 30 can control the operation by the automatic driving function of the excavator 100 while grasping the position of the tip of the attachment AT.

[0078] The excavator 100 operates an actuator (for example, a hydraulic actuator) according to the operation of an operator boarding the cab 10, and drives operating elements (hereinafter, "driven elements") such as the lower traveling body 1, the upper swing body 3, the boom 4, the arm 5, and the bucket 6.

[0079] Alternatively, instead of being configured to be operable by the operator in the cab 10, or in addition thereto, the excavator 100 may be configured to be remotely operable from outside the excavator 100. When the excavator 100 is remotely operated, the inside of the cab 10 may be unmanned.

[0080] Further, the excavator 100 may automatically operate the actuator regardless of the content of the operator's operation. Thereby, the excavator 100 realizes a function of automatically operating at least a part of the driven elements such as the lower traveling body 1, the upper swing body 3, the boom 4, the arm 5, and the bucket 6, that is, a so-called "automatic operation function" or "machine control function".

[0081] The automatic operation function may include a function of automatically operating driven elements (actuators) other than the driven element (actuator) of the operation target in response to an operation on the operator's operation device 26 or a remote operation, that is, a so-called "semi-automatic operation function" or "operation support type machine control function". Further, the automatic operation function may include a function of automatically operating at least a part of a plurality of driven elements (hydraulic actuators) on the premise that there is no operation on the operator's operation device 26 or a remote operation, that is, a so-called "fully automatic operation function" or "fully automatic type machine control function". In the excavator 100, when the fully automatic operation function is valid, the inside of the cab 10 may be unmanned. Further, the semi-automatic operation function, the fully automatic operation function, etc. may include a mode in which the operation content of the driven element (actuator) of the automatic operation target is automatically determined according to a rule defined in advance. Further, the semi-automatic operation function, the fully automatic operation function, etc. may include a mode (so-called "automatic operation function") in which the excavator 100 autonomously makes various determinations and the operation content of the driven element (hydraulic actuator) of the automatic operation target is determined autonomously according to the determination result.

[0082] Specifically, when the operator operates the arm 5 through the operating device 26, the controller 30 may automatically operate at least one of the boom 4 and the bucket 6 so that a predefined target design surface coincides with the tip position of the bucket 6. Further, the controller 30 may also automatically operate the arm 5 regardless of the operating state of the operating device 26 that operates the arm 5. That is, the controller 30 may trigger the operation of the operating device 26 by the operator and cause the attachment to perform a predefined operation. Hereinafter, the function of the controller 30 that operates at least one of the boom 4 and the bucket 6 in addition to the arm 5 according to the operation of the operating device 26 corresponding to the arm 5 is referred to as a "semiautomatic operation function". The semiautomatic operation function may be executed, for example, when a predetermined switch (hereinafter, "MC (Machine Control) switch") disposed at the tip of any of the lever devices included in the operating device 26 is operated.

[0083] The auxiliary storage device D4 stores a design data storage unit D4A, a first learned model LM1, and a second learned model LM2.

[0084] The design data storage unit D4A stores design data. The design data includes construction data indicating the three-dimensional shape after the excavator 100 has performed construction at the work site. The construction data includes position data of the construction target in the world coordinate system indicated by GNSS and three-dimensional shape data after construction. For example, the design data includes position data and three-dimensional shape data of a target design surface formed after the excavator 100 has excavated earth and sand.

[0085] The position data is expressed, for example, in the same reference coordinate system as the position data acquired by GNSS. The reference coordinate system is, for example, the World Geodetic System. The World Geodetic System is a three-dimensional orthogonal XYZ coordinate system with the origin at the center of gravity of the earth, the X-axis in the direction of the intersection of the Greenwich meridian and the equator, the Y-axis in the direction of 90 degrees east longitude, and the Z-axis in the direction of the North Pole.

[0086] The first learned model LM1 outputs from the output layer the three-dimensional shape of at least a part of the region of the object included in the measurement information, the color information of at least a part of the region of the object included in the image information, the type of the object, and information indicating the direction and distance where the object exists with respect to a predetermined position (for example, the central position) of the excavator 100, by inputting the measurement information of the space recognition device S7 and the image information of the imaging device S6.

[0087] The second learned model LM2 inputs the three-dimensional shape of at least a part of the region of the object around the working machine included in the measurement information of the space recognition device S7, the color information indicating the color of at least a part of the region of the object that appeared in the image information, and the type of the object, and complements the three-dimensional shape and color of the region not appearing in the measurement information and the image information, and outputs three-dimensional shape data (an example of three-dimensional shape information) indicating the three-dimensional shape and color of the entire object from the output layer.

[0088] In this embodiment, the case of inputting the type of the object to the second learned model LM2 will be described, but the method of inputting the type of the object is not limited. For example, the three-dimensional shape indicating a part of the region of the object and the color information indicating the color of a part of the region of the object may be input to the learned model, and three-dimensional data indicating the three-dimensional shape and color of the entire object may be output considering the type of the object inside the learned model.

[0089] As the machine learning used for generating the first learned model LM1 and the second learned model LM2, for example, a neural network may be applied, and specifically, it is machine learning using a deep neural network (DNN: Deep Neural Network), and deep learning (deep learning) may be applied. As the deep learning, for example, a convolutional neural network, RNN (Recurrent Neural Networks), or LSTM (Long Short Term Memory) may be applied.

[0090] The first learned model LM1 and the second learned model LM2 are generated by performing machine learning based on a teacher dataset generated in advance in an information processing apparatus (not shown).

[0091] Specifically, the first learned model LM1 is trained using a teacher dataset that associates measurement information and image information with the three-dimensional shape obtained by clustering point cloud data included in the measurement information for each object, the color information extracted from the regions clustered for each object included in the image information, the type of the object, and information indicating the direction and distance at which the object exists with reference to a predetermined position (e.g., the center position) of the excavator 100. That is, the first learned model LM1 can output, by means of the machine learning process, the three-dimensional shape of the object included in the measurement information, the color information extracted from the regions clustered for each object included in the image information, the type of the object, and information indicating the direction and distance at which the object exists with reference to a predetermined position (e.g., the center position) of the excavator 100.

[0092] Specifically, the second learned model LM2 is trained using a teacher dataset that associates the overall three-dimensional shape and overall color information of the object with the type of the object. That is, the second learned model LM2 can, by means of the machine learning process, complement the three-dimensional shape and color for the entire object.

[0093] Note that the second learned model LM2 may be updated by additionally training an existing second learned model LM2 with a new teacher dataset.

[0094] For example, before the excavator 100 performs work, information representing the overall three-dimensional shape and the overall color of the objects existing at the work site (an example of an area) where construction will be carried out is prepared in advance. Then, the second pre-trained model LM2 is subjected to additional machine learning processing using the teacher data based on the overall three-dimensional shape and the overall color information of the objects prepared in advance. The additional machine learning processing may be performed by the controller 30 or by another information processing device. In the present embodiment, by performing additional machine learning on the objects existing at the work site where work will be carried out, it is possible to complement the three-dimensional shape and color of the objects existing at the work site, so that an improvement in the accuracy of the complement can be realized. Note that the additional learning is not limited to the second pre-trained model LM2, and may be performed on the first pre-trained model LM1. For example, additional machine learning processing may be performed on the first pre-trained model LM1 using the three-dimensional shape obtained by clustering the objects existing at the work site and the teacher data associating the types of the objects.

[0095] The controller 30 includes an acquisition unit 301, an extraction unit 302, a complementation unit 303, a generation unit 304, and an automatic control unit 305.

[0096] The acquisition unit 301 acquires various information from various sensors. For example, the acquisition unit 301 acquires image information captured by the imaging devices S6 (front camera S6F, left camera S6L, right camera S6R, and rear camera S6B). For example, the acquisition unit 301 acquires measurement information indicating the results measured by the space recognition devices S7 (rear space recognition device S7B, left space recognition device S7L, right space recognition device S7R, and front space recognition device S7F).

[0097] The acquisition unit 301 acquires the detection information detected by each of the boom angle sensor S1, the arm angle sensor S2, the bucket angle sensor S3, the machine body inclination sensor S4, and the swing angle sensor S5. The acquisition unit 301 acquires the position and orientation of the excavator 100 from the positioning device PS.

[0098] The acquisition unit 301 acquires design data corresponding to the work site where the excavator 100 is working from the design data storage unit D4A.

[0099] The extraction unit 302 identifies the type of each object appearing in the measurement information from the measurement information and the image information acquired by the acquisition unit 301, extracts the three-dimensional shape of the region where the object exists shown in the measurement information, and extracts the color information of the region where the object exists shown in the image information.

[0100] The extraction unit 302 inputs the measurement information and the image information acquired by the acquisition unit 301 into the first pre-trained model LM1, and receives from the first pre-trained model LM1 the three-dimensional shape of at least a part of the region of the object included in the measurement information, the color information of at least a part of the region of the object included in the image information, the type of the object, and information indicating the direction and distance where the object exists with respect to a predetermined position (for example, the center position) of the excavator 100.

[0101] In this embodiment, an example using the first pre-trained model LM1 for clustering the three-dimensional shape for each object has been described. However, this embodiment is not limited to the method of using the first pre-trained model LM1 for clustering the three-dimensional shape for each object, and well-known methods may be used. Similarly, in the image information, the method of clustering (classifying) the regions for each object is not limited to the method using the first pre-trained model LM1, and well-known methods may be used. For example, based on the similarity between the data included in the point cloud data or the image information, the point cloud data or the image information may be clustered (classified) for each object.

[0102] The identification of the type of object is not limited to the method using the first learned model LM1, and it may be identified based on any one or more of the three-dimensional shape and color of the object, and well-known methods may be used. Also, a learned model other than the first learned model LM1 may be used. For example, a learned model obtained by machine learning the correspondence between the three-dimensional shape of an object and the type of the object may be generated, and when measurement information is input to the input layer of the learned model, the type of the object included in the measurement information may be output from the output layer.

[0103] FIG. 5 is a diagram showing the three-dimensional shape of an object existing around the excavator 100 based on the measurement information and the image information acquired by the acquisition unit 301 according to the present embodiment. In the example shown in FIG. 5, for ease of explanation, it represents a state in which the color of the image information is applied to the three-dimensional shape of the object shown by the measurement information.

[0104] In the example shown in FIG. 5, the measurement information measured by the space recognition device S7 is point cloud data including the three-dimensional shapes of the object 1501 and the object 1502 in the detectable region 1500.

[0105] As shown in FIG. 5, the measurement information includes the three-dimensional shapes of the object 1501 and the object 1502. When the space recognition device S7 is a LIDAR, the three-dimensional shapes of the object 1501 and the object 1502 included in the measurement information are limited to the three-dimensional shape of the region where the signal (light) irradiated from the space recognition device S7 reaches. That is, among the objects (for example, the object 1501 and the object 1502), the region on the opposite side of the region facing the space recognition device S7 is a region that is not measured because the signal (light) irradiated from the space recognition device S7 does not reach. In other words, a part of the three-dimensional shape of the object that can be referred to from the excavator 100 provided with the space recognition device S7 is included in the measurement information, and a part of the object that cannot be referred to from the excavator 100 is a region that is not measured.

[0106] Note that this embodiment does not limit the unmeasured area to the area on the side opposite to the area facing the spatial recognition device S7. For example, even in the area facing the spatial recognition device S7, when the object has a complex configuration, while the three-dimensional shape of the first area reached by the light (signal) irradiated from the spatial recognition device S7 is measured, there may be a second area blocked by the light (signal) by the first area reached by the light (signal). In this case, the second area where the light (signal) irradiated from the spatial recognition device S7 did not reach becomes an unmeasured area. Furthermore, in the area where the object has a complex configuration, due to the relationship of resolution, the spatial recognition device S7 may not recognize the complex configuration, and there is a possibility of including an unmeasured area. Also, depending on the color or material of the object, the spatial recognition device S7 may not recognize the signal (light) reflected from the object, and there is a possibility of becoming an unmeasured area. Therefore, in this embodiment, in the complementing unit 303 described later, the three-dimensional shape of the unmeasured area is complemented.

[0107] FIG. 6 is a diagram showing the three-dimensional shape of an object extracted by the extraction unit 302 from the three-dimensional shape shown in the measurement information acquired by the acquisition unit 301.

[0108] The extraction unit 302 uses the first learned model LM1 to extract, from the measurement information and the image information, for each of the object 1501 and the object 1502, the three-dimensional shape, color information, object type, and information indicating the direction and distance at which the object exists with respect to a predetermined position (for example, the center position) of the shovel 100.

[0109] Specifically, the extraction unit 302 inputs the measurement information and the image information into the first learned model LM1, and among the measurement information, the three-dimensional shape of the area 1601 represented by the object clustered as the object 1501, and among the image information, the color information included in the area 1601 represented by the object, the object type of the object 1501, and information 1611 indicating the direction and distance at which the area 1601 exists with respect to a predetermined position (for example, the center position) of the shovel 100. In the example shown in FIG. 6, information (for example, a character string) indicating a shovel is received as the type of the object 1501.

[0110] Furthermore, the extraction unit 302 inputs the measurement information and the image information into the first learned model LM1, and thereby receives, among the measurement information, the three-dimensional shape of the region 1602 representing the object that is clustered as the object 1502, among the image information, the color information included in the region 1602 representing the object, the type of the object 1502, and information 1612 indicating the direction and distance in which the region 1602 exists with reference to a predetermined position (for example, the central position) of the excavator 100. In the example shown in FIG. 6, as the type of the object 1502, information (for example, a character string) indicating a dump truck is received.

[0111] The complementing unit 303 is configured to complement the three-dimensional shape of the region of the object that has not been measured by the space recognition device S7, based on the three-dimensional shape of at least a partial region of the object included in the measurement information measured by the space recognition device S7. Furthermore, the complementing unit 303 is configured to complement the color of the region of the object that has not been imaged by the imaging device S6, based on the color information of at least a partial region of the object in the image information imaged by the imaging device S6.

[0112] As described above, the region that has not been measured by the space recognition device S7 includes the region existing on the opposite side of the region facing the space recognition device S7 among the regions representing the surface of the object. In the present embodiment, by complementing the three-dimensional shape of the region that has not been measured by the space recognition device S7, the region and shape in which the object exists can be grasped, so that the work considering the object becomes easy, and thus an improvement in work efficiency can be realized.

[0113] The completion unit 303 receives the three-dimensional shape and color information extracted as a partial region of the object, and the type of the object, inputs them to the second pre-trained model LM2, and receives from the second pre-trained model LM2 the three-dimensional shape data of the object in which the three-dimensional shape and color of the region not shown in the measurement information and the image information are completed. Since the second pre-trained model LM2 according to the present embodiment uses the three-dimensional shape and color of an actual object as teacher data, improvement in the accuracy of completion can be realized.

[0114] FIG. 7 is a diagram showing the concept of the process performed by the completion unit 303 according to the present embodiment.

[0115] For example, the completion unit 303 inputs the three-dimensional shape and color information of the region 1601 and the type of the object "excavator" to the second pre-trained model LM2, and thereby the three-dimensional shape and color of the region not measured by the spatial recognition device S7 and the imaging device S6 in the three-dimensional shape of the region 1601 are completed as an excavator, and receives the three-dimensional shape data 1701.

[0116] For example, the completion unit 303 inputs the three-dimensional shape and color information of the region 1602 and the type of the object "dump truck" to the second pre-trained model LM2, and thereby the three-dimensional shape and color of the region not measured by the spatial recognition device S7 and the imaging device S6 in the three-dimensional shape of the region 1602 are completed as a dump truck, and receives the three-dimensional shape data 1702.

[0117] The three-dimensional shape data output from the second pre-trained model LM2 is, for example, information representing the three-dimensional shape of the entire object in the form of a point cloud or a mesh, and is data in which each point in the point cloud is colored. Note that the present embodiment shows an example of the three-dimensional shape data, and any data that can recognize the three-dimensional shape of the object may be used.

[0118] In this embodiment, as an example of an object to be complemented, an example of complementing the three-dimensional shape and color of a shovel and a dump truck has been described. This embodiment does not limit the object to be complemented to a shovel and a dump truck, but enables various objects to be complemented. For example, the complementing unit 303 according to this embodiment complements, as an object in an area not measured by the space recognition device S7 and the imaging device S6, any one or more of a working machine including a shovel, a construction machine, a dump truck, a building, and a structure. By complementing the areas not measured for each object existing at the work site in this embodiment, the three-dimensional shape of each object existing at the work site can be clarified, so it becomes easy to grasp the objects existing at the work site and the situation of the objects.

[0119] That is, the second learned model LM2 has been machine-learned for any one or more of a working machine including a shovel, a construction machine, a dump truck, a building, and a structure as the type of object. Note that this embodiment does not limit the method of complementing the three-dimensional shape and color of all types of objects with a single second learned model LM2. For example, a learned model may be provided for each type of object.

[0120] In this embodiment, an example of using a method for complementing the three-dimensional shape measured in the measurement information will be described. By the way, there is a technology in which a controller of a working machine stores a three-dimensional shape model for each object in advance, and virtually realizes the surrounding situation by arranging the three-dimensional shape for each object stored in advance based on the measurement result. However, in this method, there is a possibility that the three-dimensional shapes of objects of different types from the objects existing in the real space are arranged in the virtual three-dimensional space. For example, although a 4t dump truck exists in the real space, the controller of the working machine may arrange the three-dimensional shape model of a 10t dump truck in the virtual three-dimensional space. In this case, when the controller of the working machine performs an operation of loading an object on a 4t dump truck existing in the real space, there may be a deviation between the loadable weight and the actually loaded weight. Furthermore, when the controller sets the movement path of the working machine so as to avoid the 4t dump truck, there is a possibility that an appropriate movement path cannot be set.

[0121] Therefore, the controller 30 according to this embodiment uses the second learned model LM2 to complement the unmeasured areas of the three-dimensional shape for each object included in the measurement information. By this method, the use of three-dimensional shapes different from the objects actually existing in the real space is suppressed, so that operations more suitable for the surrounding situation become possible.

[0122] The generation unit 304 generates information on a virtual three-dimensional space showing the surrounding situation of the excavator 100 using the three-dimensional shape data for each object received from the complementation unit 303.

[0123] The virtual three-dimensional space is, for example, a virtual three-dimensional space with the center position of the excavator 100 as the origin, in which the three-dimensional shapes of the land and objects existing around the excavator 100 are shown, and the ground and objects existing around the excavator 100 are colored. The ground existing around the excavator 100 according to the present embodiment may be the ground at the work site of the excavator 100, and includes, for example, the ground where construction is performed (including slopes), the road surface for traveling, and the slope surface after construction, etc.

[0124] In order to generate the information of the virtual three-dimensional space, the generation unit 304 acquires the information indicating the three-dimensional shape and color of the ground around the excavator 100. The information indicating the three-dimensional shape and color of the ground around the excavator 100 may be arbitrary information, and may be information measured from the work site using the space recognition device S7, the imaging device S6, etc. before the work, or may be information held as construction data. Furthermore, the information indicating the three-dimensional shape and color of the ground around the excavator 100 may be based on the information acquired from the space recognition device S7 and the imaging device S6 of the excavator 100, or may be information with the three-dimensional shape and color complemented as necessary.

[0125] Then, the generation unit 304 generates the information of the virtual three-dimensional space by arranging the complemented three-dimensional shape data of the object on the three-dimensional shape (colored) indicating the ground around the excavator 100.

[0126] FIG. 8 is a conceptual diagram showing the information of the virtual three-dimensional space generated by the generation unit 304 according to the present embodiment. In the example shown in FIG. 8, the three-dimensional shape and color of the ground around the excavator 100 are represented in the region 1801. The positional relationship between the ground around the excavator 100 and the excavator 100 can be specified based on the reference coordinate system.

[0127] Furthermore, the generation unit 304 arranges the three-dimensional shape data for each object based on the direction and distance where the object exists with respect to the reference of the excavator 100.

[0128] For example, based on a predetermined position (e.g., the central position) 1850 of the excavator 100, the generation unit 304 arranges the three-dimensional shape data 1701 of the excavator according to the information 1611 indicating the direction and distance in which the region 1601 extracted by the extraction unit 302 exists.

[0129] Furthermore, based on a predetermined position (e.g., the central position) 1850 of the excavator 100, the generation unit 304 arranges the three-dimensional shape data 1702 of the dump truck according to the information 1612 indicating the direction and distance in which the region 1602 extracted by the extraction unit 302 exists.

[0130] By referring to the virtual three-dimensional space generated in the above-described process, the controller 30 according to the present embodiment can recognize the positional relationship between the ground and objects around the excavator 100. Therefore, since the controller 30 can perform various operations considering the positional relationship, improvement in work efficiency can be realized.

[0131] In addition, since the three-dimensional shape of the object according to the present embodiment uses the three-dimensional shape of the object included in the measurement information measured by the space recognition device S7, the current movement of the object is reflected. That is, the controller 30 can estimate the current operation of the object by recognizing the three-dimensional shape of the object shown in the virtual three-dimensional space. Therefore, the controller 30 can perform operations considering the current operations of the objects existing around.

[0132] Based on the design data stored in the design data storage unit D4A and the information of the virtual three-dimensional space generated by the generation unit 304, the automatic control unit 305 autonomously operates the excavator 100.

[0133] The automatic control unit 305 generates a trajectory for operating the attachment AT or the lower traveling body 1, and outputs a control command (current command) to the proportional valve 31 so as to operate the attachment AT or the lower traveling body 1 according to the trajectory. The proportional valve 31 autonomously operates each actuator by individually adjusting the pilot pressure acting on the control valve corresponding to each actuator according to the control command. Note that the method of generating the trajectory is not described as it may be any well-known method.

[0134] For example, when the automatic control unit 305 recognizes that the dump truck is stopped in a standby state near the excavator 100 with reference to the virtual three-dimensional space, the automatic control unit 305 operates the attachment AT and the upper swing body 3 to discharge soil onto the loading platform of the dump truck.

[0135] As another example, the automatic control unit 305 generates a movement path of the attachment AT so as to form the target design surface shown in the design data while suppressing contact with an object represented in a (complemented) three-dimensional shape in the virtual three-dimensional space. Then, the automatic control unit 305 operates the attachment AT to follow the movement path. Thereby, the attachment AT operates so as to form the target design surface while suppressing contact with the object.

[0136] Furthermore, the automatic control unit 305 suppresses contact with an object represented in a (complemented) three-dimensional shape in the virtual three-dimensional space and generates a movement path so as to travel on the ground represented in the virtual three-dimensional space in order to move to the destination shown in the design data. Then, the automatic control unit 305 operates the lower traveling body 1 to travel according to the movement path. Thereby, the excavator 100 can move to the destination while suppressing contact with the object.

[0137] In this way, by using the information of the virtual three-dimensional space generated by the generation unit 304, the automatic control unit 305 can recognize the exact area and size occupied by the objects existing around the excavator 100, etc., so that it is possible to perform operations or driving, etc. considering the current state of the objects.

[0138] Next, the processing procedure executed by the controller 30 according to the present embodiment will be described. FIG. 9 is a flowchart showing the processing procedure until the controller 30 according to the present embodiment performs automatic control.

[0139] First, the acquisition unit 301 acquires the image information captured by the imaging device S6 and the measurement information measured by the space recognition device S7 (S1901).

[0140] The extraction unit 302 inputs the measurement information and the image information acquired by the acquisition unit 301 into the first learned model LM1, and receives from the first learned model LM1, for each object represented in the measurement information and the image information, the type of the object, the three-dimensional shape of at least a part of the region of the object, the color information of at least a part of the region of the object, and the information indicating the direction and distance where the object exists (S1902).

[0141] The complementation unit 303 inputs the three-dimensional shape, the color information, and the type of the object for each object into the second learned model LM2, and receives from the second learned model LM2 the three-dimensional shape data of the entire object in which the three-dimensional shape and the color of the regions not represented in the measurement information and the image information are complemented (S1903).

[0142] The generation unit 304 generates information of a virtual three-dimensional space showing the positional relationship between the objects and the ground around the excavator 100 based on the three-dimensional shape data of each object complemented by the complementation unit 303 and the three-dimensional shape and color of the ground around the excavator 100 (S1904).

[0143] The automatic control unit 305 performs automatic control of the excavator 100 based on the design data stored in the design data storage unit D4A and the information on the virtual three-dimensional space generated by the generation unit 304 (S1905).

[0144] In the present embodiment, since the controller 30 has the above-described configuration, it can recognize the three-dimensional shape of an object around the excavator 100, enabling automatic control considering the three-dimensional shape of the object.

[0145] In the present embodiment, the case where the controller 30 performs the above-described processing has been described. However, the present embodiment is not limited to the method by which the controller 30 performs the above-described processing. For example, the controller 30 may transmit image information and measurement information to an external server via the communication device T1. Then, the external server performs the above-described processing, complements the three-dimensional shape and color of the ground around the excavator 100, and may transmit information on the virtual three-dimensional space showing the positional relationship between the objects and the ground around the excavator 100 to the communication device T1. Then, the controller 30 may autonomously operate the excavator 100 based on the received information on the virtual three-dimensional space.

[0146] Also, in the present embodiment, a method for autonomously controlling the excavator 100 based on information on the virtual three-dimensional space has been described. However, the present embodiment does not limit the method for autonomously controlling the excavator 100. For example, when the semi-automatic operation function is executed by an operator's operation of a predetermined switch, the controller 30 may operate the attachment AT or the lower traveling body 1 based on the information on the virtual three-dimensional space. As another example, the first display device D3 may display the information on the virtual three-dimensional space generated by the generation unit 304. Then, the operator may operate the excavator 100 after checking the surrounding situation with reference to the information displayed on the first display device D3.

[0147] (Modification example) The above-described embodiment has been described with an example of performing automatic control of the excavator 100 based on the measurement information of the space recognition device S7 and the image information of the imaging device S6. However, the above-described embodiment does not limit the method using the measurement information of the space recognition device S7 and the image information of the imaging device S6. Therefore, as a modification example, the case of using the measurement information of the space recognition device S7 will be described. The second learned model LM2 according to this modification example is subjected to machine learning processing so as to output three-dimensional shape data of an object in which the three-dimensional shape of a region that has not been measured is complemented when the type and three-dimensional shape of the object are input.

[0148] The extraction unit 302 according to this modification example identifies the type of the object included in the measurement information and extracts the three-dimensional shape of the object.

[0149] The complementation unit 303 according to this modification example inputs the type of the object and the three-dimensional shape of the object to the second learned model LM2, and receives three-dimensional shape data of the entire object in which the region not measured by the space recognition device S7 is complemented.

[0150] The generation unit 304 according to this modification example generates information on a virtual three-dimensional space showing the positional relationship between the objects and the ground around the excavator 100 based on the three-dimensional shape data of the object complemented by the complementation unit 303.

[0151] The automatic control unit 305 according to this modification example autonomously operates the excavator 100 based on the design data stored in the design data storage unit D4A and the information on the virtual three-dimensional space generated by the generation unit 304.

[0152] The controller 30 according to this modification example can recognize the three-dimensional shape of the objects around the excavator 100 based on the measurement information measured by the space recognition device S7 without using the image information captured by the imaging device S6. Therefore, while having the same effects as the above-described embodiment, cost reduction can be achieved.

[0153] (Second Embodiment) In the above-described embodiments and modifications, the case where the controller 30 of the excavator 100 recognizes the surrounding situation and performs automatic control has been described. However, the above-described embodiments and modifications do not limit the recognition of the surrounding situation for performing automatic control. Therefore, in the present embodiment, the case of monitoring the periphery of the excavator 100 will be described.

[0154] FIG. 10 is a schematic diagram showing a configuration example of a support system SYS of the excavator 100 according to the present embodiment. FIG. 10 shows an example in which the excavator 100, the fixed-point measurement device 2150, the information center 2100, and the remote operation room RC are connected.

[0155] The excavator 100 uses a communication device T1 provided in the excavator 100 to transmit the detection results from various sensors provided in the excavator 100 to the information center 2100. For example, the excavator 100 transmits the image information captured by the imaging device S6, the measurement information measured by the space recognition device S7, and the position information of the positioning device PS to the information center 2100.

[0156] Further, the above-described embodiment does not limit the method of providing the imaging device S6 and the space recognition device S7 in the excavator 100. Therefore, in the present embodiment, an example in which the fixed-point measurement device 2150 is provided at the work site will be described. As an example different from the above-described embodiment, when the fixed-point measurement device 2150 is provided at the work site, a mode in which the imaging device S6 and the space recognition device S7 are not mounted on the excavator 100 is also conceivable.

[0157] One or more fixed-point measurement devices 2150 are installed at the work site where the excavator 100 performs work. The fixed-point measurement device 2150 functions as the imaging device S6 and the space recognition device S7 described in the above-described embodiment. For example, the fixed-point measurement device 2150 transmits the image information and the measurement information to the information center 2100.

[0158] The information center 2100 according to this embodiment has the same configuration as the controller 30 of the above-described embodiment. The information center 2100 receives image information and measurement information from one or more of the excavator 100 and the fixed-point measurement device 2150.

[0159] The information center 2100 may combine and process the image information and measurement information received from the excavator 100 and the image information and measurement information received from the fixed-point measurement device 2150. In the virtual three-dimensional space, the information center 2100 may generate three-dimensional shape data by arranging the three-dimensional shape and color for each object based on each of the plurality of pieces of image information and measurement information, and then complementing the three-dimensional shape and color of the area that has not been measured in the object.

[0160] Furthermore, the information center 2100 may generate three-dimensional shape data showing the three-dimensional shape and color of the ground at the current work site based on the image information and measurement information received from one or more of the excavator 100 and the fixed-point measurement device 2150.

[0161] Then, the information center 2100 generates information on the virtual three-dimensional space representing the work site. The information on the virtual three-dimensional space includes the three-dimensional shape of the work machine (e.g., excavator) existing at the work site and the three-dimensional shape of the ground at the work site.

[0162] Through the above-described control, the information center 2100 arranges the three-dimensional shape of the object existing around the excavator 100 in the virtual three-dimensional space after complementing the area that has not been measured.

[0163] Then, the information center 2100 displays the information on the virtual three-dimensional space on the display device 2110.

[0164] The administrator of the information center 2100 can refer to the virtual three-dimensional space from any perspective according to the operation. That is, since the three-dimensional shape of the objects arranged in the virtual three-dimensional space is complemented even in the areas that have not been measured, the administrator can recognize the three-dimensional shape of the entire object existing at the work site from any perspective.

[0165] As a result, the administrator of the information center 2100 can recognize the current progress status of any one or more of the working machines, construction machines, dump trucks, buildings, and structures existing at the work site. Since the administrator of the work site can grasp the current situation of the objects at the work site, the progress of the construction can be managed and the work can be carried out according to the plan.

[0166] Furthermore, the information center 2100 may transmit the information of the virtual three-dimensional space to the excavator 100.

[0167] In this case, the excavator 100 performs automatic control based on the received information of the virtual three-dimensional space and the design data.

[0168] Furthermore, the excavator 100 according to the present embodiment may be remotely operated. In the support system SYS according to the present embodiment, a remote operation room RC is provided. The remote operation room RC is provided with a display device DR, an operation device R26, an operation sensor R29, an operation seat DS, and a remote controller R30, and a communication device T2.

[0169] The remote controller R30 may receive the information of the virtual three-dimensional space via the communication device T2 and display the information of the virtual three-dimensional space on the display device DR.

[0170] Furthermore, the remote controller R30 may have the same functions as the controller 30 in the above-described embodiment. In this case, the remote controller R30 may receive image information and measurement information via the information center 2100, perform the same processing as the controller 30, generate information on the virtual three-dimensional space, and display the information on the virtual three-dimensional space on the display device DR. Thereby, the operator OP present in the operator's seat DS can check the situation around the excavator 100 even when present in the remote operation room RC.

[0171] The operator OP present in the operator's seat DS in the remote operation room RC checks the surrounding situation and operates the operating device R26 as necessary. Then, the operation sensor R29 detects the operation content received by the operating device R26.

[0172] Then, the remote controller R30 generates a control signal for operating the excavator 100 based on the detected operation content according to the detected operation content. Then, the communication device T2 transmits the generated control signal to the excavator 100. By transmitting the control signal, the remote controller R30 can control the excavator 100.

[0173] In the present embodiment, the excavator 100 may perform automatic control in the same manner as in the first embodiment. Then, when the operator OP present in the remote operation room RC monitors the current situation at the work site and determines that it is better to operate by himself / herself, the remote controller R30 may switch the control so that the excavator 100 is operated by the operating device R26 in response to the operation from the operator OP.

[0174] In the present embodiment, when a manager or the like manages the work site, the situation of the work site can be recognized in a state where the three-dimensional shape of an object in an area that has not been measured is complemented, so that the visibility is improved and it becomes easy to grasp the current situation.

[0175] <Function> In the above-described embodiments and modifications, the controller 30, the information center 2100, or the remote controller R30 performs the above-described control to complement the three-dimensional shape of an object in an area that cannot be measured, thereby enabling recognition of the three-dimensional shape of an object existing around the excavator 100. Then, since the excavator 100 can perform work in consideration of the three-dimensional shape of the object, it can perform movement control and the like in consideration of the area where the object exists, thus realizing an improvement in work efficiency.

[0176] As described above, the embodiments of the system for a working machine and the excavator according to the present invention have been described. However, the present invention is not limited to the above-described embodiments. Within the scope described in the claims, various changes, modifications, substitutions, additions, deletions, and combinations are possible. Naturally, they also belong to the technical scope of the present invention.

Explanation of Reference Numerals

[0177] 100 Excavator 1 Lower Traveling Body 2 Swing Mechanism 3 Upper Swing Structure 4 Boom 5 Arm 6 Bucket S1 Boom Angle Sensor S2 Arm Angle Sensor S3 Bucket Angle Sensor S4 Machine Body Tilt Sensor S5 Swing Angle Sensor S6 Imaging Device S7 Spatial Recognition Device PS Positioning Device T1 Communication Device D4 Auxiliary Storage Device D4A Design Data Storage Unit LM1 First Learned Model LM2 Second Learned Model 30 Controller 301 Acquisition Unit 302 Extraction Unit 303 Completion Unit 304 Generation Unit 305 Automatic Control Unit RC Remote Operation Room R30 Remote Controller T2 Communication Device 2100 Information Center 2110 Display Device 2150 Fixed-Point Measuring Device

Claims

1. A spatial recognition device capable of measuring the three-dimensional shape of an object existing around a working machine, and A control device configured to complement the three-dimensional shape of a region of the object that has not been measured by the spatial recognition device based on the three-dimensional shape of at least a part of the region of the object included in the measurement information measured by the spatial recognition device, A system for a working machine comprising the above.

2. The object for which the control device complements the unmeasured region is any one or more of a working machine including a shovel, a construction machine, a dump truck, a building, and a structure, The system for a working machine according to claim 1.

3. The unmeasured region is a region existing on the side opposite to the region facing the spatial recognition device among the regions representing the surface of the object, The system for a working machine according to claim 1.

4. Further comprising a learned model obtained by performing machine learning processing using teacher data showing the three-dimensional shape of the entire object, The control device receives the three-dimensional shape information of the object in which the three-dimensional shape of the region that has not been measured by the spatial recognition device is complemented by inputting the three-dimensional shape of at least a part of the region of the object included in the measurement information into the learned model. The system for a working machine according to claim 1.

5. The learned model has been subjected to machine learning processing using the teacher data showing the three-dimensional shape of the object existing in the area where construction is performed by the working machine. The system for a working machine according to claim 4.

6. Further comprising an imaging device for imaging the periphery of the working machine, The learned model has been subjected to machine learning processing using the teacher data showing the overall color of the object, The control device receives the three-dimensional shape information in which the color is assigned and the color of the region not shown in the image information is complemented by further inputting the color information representing the color of a part of the region of the object shown in the image information captured by the imaging device into the learned model. The system for a working machine according to claim 4.

7. Further comprising a learned model obtained by performing machine learning processing using teacher data showing the three-dimensional shape and the type of the object, The control device inputs the measurement information including the three-dimensional shape of the object into the learned model, and receives the type of the object included in the measurement information. The control device is configured to complement the three-dimensional shape of the region of the object that has not been measured by the space recognition device, based on the three-dimensional shape of at least a part of the region of the object and the type of the object included in the measurement information. The system for a working machine according to claim 1.

8. The control device generates information on a three-dimensional space in which the three-dimensional shape of the complemented object is arranged with respect to the three-dimensional shape of the ground around the working machine. The system for a working machine according to claim 1.

9. The space recognition device is attached to the working machine or installed at a work site where the working machine performs work. The system for a working machine according to claim 1.

10. A lower traveling body An upper swing body that is swingably mounted on the lower traveling body A space recognition device attached to the upper swing body and capable of measuring the three-dimensional shape of an object existing around the excavator A control device configured to complement the three-dimensional shape of the region of the object that has not been measured by the space recognition device, based on the three-dimensional shape of at least a part of the region of the object included in the measurement information measured by the space recognition device An excavator comprising the same.

Images