System for work machine and excavator

Abstract

To improve safety.SOLUTION: A system for a work machine according to one aspect of the present disclosure includes: a spatial recognition device capable of measuring shapes of objects present around the work machine; and a control device configured to identify shapes of other work machines present around the work machine, for each part that constitutes the other work machine, based on measurement information representing measurement results obtained by the spatial recognition device.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a system for a working machine and an excavator.

Background Art

[0002] Conventionally, a space recognition device is often provided to detect an object existing around a working machine (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Patent Document 1 describes detecting an object existing around an excavator and calculating the distance to the object. However, it does not detect up to the parts constituting the object. That is, when another working machine is detected at a work site, since the parts of the working machine are not recognized, it is difficult to grasp the current situation of the working machine.

[0005] One aspect of the present invention proposes a technique for improving safety by identifying the parts of another working machine, recognizing the current situation of the other working machine, and enabling work according to the current situation.

Means for Solving the Problems

[0006] A system for a working machine according to one aspect of the present invention includes a space recognition device capable of measuring the shape of an object existing around the working machine, and based on measurement information representing the measurement result by the space recognition device, the shape of another working machine existing around the working machine is identified for each part constituting the other working machine, and a control device configured to perform the identification.

Advantages of the Invention

[0007] According to one aspect of the present invention, by enabling work according to the current situation, an improvement in safety is realized.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Modes for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Also, the embodiments described below are illustrative rather than limiting the invention, and not all features and combinations thereof described in the embodiments are necessarily essential to the invention. In each drawing, the same or corresponding components are denoted by the same or corresponding reference numerals, and the description thereof may be omitted.

[0010] Hereinafter, in an embodiment of the present invention, an example of using an excavator as an example of a working machine will be described, but the present invention is not limited to excavators. It may be applied to construction machines, standard machines, application machines, forestry machines, or transport machines based on hydraulic excavators.

[0011] (First Embodiment) First, with reference to FIG. 1, an overview of the excavator 100 according to the present embodiment will be described. FIG. 1 is a side view of the excavator 100 according to the first embodiment. FIG. 2 is a top view of the excavator 100 according to the first embodiment.

[0012] An upper swing body 3 is swingably mounted on a lower traveling body 1 of the excavator 100 via a swing mechanism 2. A boom 4 is attached to the upper swing body 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 as an end attachment is attached to the tip of the arm 5. The end attachment may be a slope bucket, a dredging bucket, or the like.

[0013] The boom 4, the arm 5, and the bucket 6 constitute an excavation attachment which is an example of the attachment AT, and are respectively hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9. A boom angle sensor S1 is attached to the boom 4, an arm angle sensor S2 is attached to the arm 5, and a bucket angle sensor S3 is attached to the bucket 6. The excavation attachment may be provided with a bucket tilt mechanism.

[0014] The boom angle sensor S1 detects the rotation angle of the boom 4. In this embodiment, the boom angle sensor S1 is an acceleration sensor and can detect the boom angle, which is the rotation angle of the boom 4 with respect to the upper slewing body 3. The boom angle, for example, becomes the minimum angle when the boom 4 is lowered to the lowest position, and increases as the boom 4 is raised.

[0015] The boom angle sensor S1 may include, for example, a rotary encoder, an acceleration sensor, a six-axis sensor, an IMU (Inertial Measurement Unit), etc. Further, the boom angle sensor S1 may also include a potentiometer using a variable resistor, a cylinder stroke sensor that detects the stroke amount of a hydraulic cylinder (boom cylinder 7) corresponding to the boom angle, etc. The same applies to the arm angle sensor S2, the bucket angle sensor S3, and the machine body inclination sensor S4 below. The detection signal corresponding to the boom angle by the boom angle sensor S1 is taken into the controller 30.

[0016] The arm angle sensor S2 detects the rotation angle of the arm 5. In this embodiment, the arm angle sensor S2 is an acceleration sensor and can detect the arm angle, which is the rotation angle of the arm 5 with respect to the boom 4. The arm angle, for example, becomes the minimum angle when the arm 5 is closed to the maximum extent, and increases as the arm 5 is opened.

[0017] The bucket angle sensor S3 detects the rotation angle of the bucket 6. In this embodiment, the bucket angle sensor S3 is an acceleration sensor and can detect the bucket angle, which is the rotation angle of the bucket 6 with respect to the arm 5. The bucket angle, for example, becomes the minimum angle when the bucket 6 is closed to the maximum extent, and increases as the bucket 6 is opened.

[0018] The boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 may be a potentiometer using a variable resistor, a stroke sensor that detects the stroke amount of the corresponding hydraulic cylinder, or a rotary encoder that detects the rotation angle around the connecting pin, etc. The boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 constitute an attitude sensor that detects the attitude of the excavation attachment.

[0019] The upper swing body 3 is provided with a cabin 10 which is a driver's cab and is equipped with a power source such as an engine 11. Further, a body tilt sensor S4, a swing angle sensor S5, an imaging device S6, and a space recognition device S7 are attached to the upper swing body 3. Further, a communication device T1 and a positioning device PS are attached to the upper swing body 3.

[0020] The body tilt sensor S4 is configured to detect the tilt of the upper swing body 3 with respect to a predetermined plane. In the present embodiment, the body tilt sensor S4 is an acceleration sensor that detects the tilt angle around the front-rear axis and the tilt angle around the left-right axis of the upper swing body 3 with respect to the horizontal plane. The front-rear axis and the left-right axis of the upper swing body 3 pass through, for example, a point on the swing axis of the excavator 100 that is the excavator center point and are orthogonal to each other.

[0021] The swing angle sensor S5 is configured to detect the swing angular velocity of the upper swing body 3. In the present embodiment, the swing angle sensor S5 is a gyro sensor. The swing angle sensor S5 may be a resolver or a rotary encoder, etc. The swing angle sensor S5 may detect the swing speed. The swing speed may be calculated from the swing angular velocity.

[0022] In addition, when the body tilt sensor S4 includes a gyro sensor, a 6-axis sensor, an IMU, etc. that can detect the angular velocity around three axes, the swing state (for example, the swing angular velocity) of the upper swing body 3 may be detected based on the detection signal of the body tilt sensor S4. In this case, the swing angle sensor S5 may be omitted.

[0023] The imaging device S6 is configured to acquire images of the surroundings of the excavator 100. In the present embodiment, the imaging device S6 includes a front camera S6F that images the space in front of the excavator 100, a left camera S6L that images the space to the left of the excavator 100, a right camera S6R that images the space to the right of the excavator 100, and a rear camera S6B that images the space behind the excavator 100.

[0024] The imaging device S6 is, for example, a monocular camera having an imaging element such as a CCD or a CMOS, and outputs the captured image to the first display device D3 via the controller 30.

[0025] The input device D2 receives operation inputs from the operator and outputs them to the controller 30. The input device D2 includes, for example, any hardware operation means such as a touch panel, a touch pad, buttons, toggles, and rotary knobs. Further, the input device D2 may include software operation means that can be operated through hardware operation means such as virtual button icons on an operation screen displayed on the first display device D3 or the like.

[0026] As shown in FIG. 2, the front camera S6F is attached to, for example, the roof of the cab 10. The left camera S6L is attached to the left end of the upper slewing body 3. The right camera S6R is attached to the right end of the upper slewing body 3. The rear camera S6B is attached to the rear end of the upper slewing body 3.

[0027] Note that the two-dot chain line in FIG. 2 represents the viewing angle (imaging range) in the top view of each of the imaging devices S6. Specifically, the imaging range CZ includes the imaging range CZB of the rear camera S6B, the imaging range CZF of the front camera S6F, the imaging range CZL of the left camera S6L, and the imaging range CZR of the right camera S6R. The four monocular cameras are preferably attached to the upper slewing body 3 so as not to protrude from the upper surface of the upper slewing body 3 as shown in FIG. 2.

[0028] In the present embodiment, by providing the imaging device S6 in the above-described arrangement, it is possible to image an object existing around the excavator 100.

[0029] The space recognition device S7 is configured to recognize the state of the space around the excavator 100. The space recognition device S7 includes a rear space recognition device S7B that detects the space behind the excavator 100, a left space recognition device S7L that detects the space to the left of the excavator 100, a right space recognition device S7R that detects the space to the right of the excavator 100, and a front space recognition device S7F that detects the space in front of the excavator 100.

[0030] The space recognition device S7 may use LIDAR to detect objects existing around the excavator 100. LIDAR measures, for example, the distance between more than one million points within the monitoring range and the LIDAR. Note that this embodiment is not limited to the method using LIDAR, and any space recognition device capable of measuring the distance to an object may be used. For example, a stereo camera may be used, or a ranging device such as a distance image camera or a millimeter-wave radar may be used. When a millimeter-wave radar or the like is used as the space recognition device S7, a large number of signals (such as laser light) may be transmitted from the space recognition device S7 toward an object, and the distance and direction of the object may be derived from the reflected signal by receiving the reflected signal.

[0031] The rear space recognition device S7B is attached to the rear end of the upper slewing body 3's upper surface. The left space recognition device S7L is attached to the left end of the upper slewing body 3's upper surface. The right space recognition device S7R is attached to the right end of the upper slewing body 3's upper surface. The front space recognition device S7F is attached to the front end of the cab 10's upper surface.

[0032] Note that the dotted lines in Fig. 2 represent the viewing angles (monitoring ranges) of each of the space recognition device S7 in top view. Specifically, it includes the imaging range OZB of the rear space recognition device S7B, the imaging range OZF of the front space recognition device S7F, the imaging range OZL of the left space recognition device S7L, and the imaging range OZR of the right space recognition device S7R. The four monocular cameras are preferably attached to the upper slewing body 3 so as not to protrude from the upper surface of the upper slewing body 3 as shown in Fig. 2.

[0033] The space recognition device S7 may be configured to detect a predetermined object within a predetermined area set around the excavator 100. For example, the space recognition device S7 may have a human detection function configured to detect a human while distinguishing between a human and an object other than a human.

[0034] The communication device T1 is a device that controls communication between the excavator 100 and the outside. The communication device T1 controls, for example, wireless communication between an external GNSS (Global Navigation Satellite System) surveying system and the excavator 100. The excavator 100 can acquire design data via wireless communication by using the communication device T1. However, the excavator 100 may acquire design data by using a semiconductor memory or the like. Note that the design data includes three-dimensional design data.

[0035] The positioning device PS is configured to acquire information regarding the position of the excavator 100. In the present embodiment, the positioning device PS is configured to measure the position and orientation of the excavator 100. Specifically, the positioning device PS is a GNSS receiver incorporating an electronic compass, and measures the latitude, longitude, and altitude of the current position of the excavator 100, and also measures the orientation of the excavator 100.

[0036] FIG. 3 is a diagram showing a configuration example of the drive control system of the excavator 100 in FIG. 1. In FIG. 3, the mechanical power transmission system is shown by a double line, the hydraulic line is shown by a thick solid line, the pilot line is shown by a broken line, and the electric drive / control system is shown by a thin solid line.

[0037] The engine 11 is a power source of the excavator 100. In the present embodiment, the engine 11 is a diesel engine that employs isochronous control to maintain a constant engine speed regardless of an increase or decrease in the engine load. The fuel injection amount, fuel injection timing, boost pressure, etc. in the engine 11 are controlled by an engine control unit (ECU) D7.

[0038] The rotating shafts of the main pump 14 and the pilot pump 15, each serving as a hydraulic pump, are connected to the rotating shaft of the engine 11. The control valve unit 17 is connected to the main pump 14 via an operating oil line.

[0039] The control valve unit 17 is a hydraulic control device that controls the hydraulic system of the excavator 100. Hydraulic actuators such as the left and right travel hydraulic motors, the boom cylinder 7, the arm cylinder 8, the bucket cylinder 9, and the swing hydraulic motor are connected to the control valve unit 17 via operating oil lines. The swing hydraulic motor may be a swing electric generator.

[0040] Figure 3 shows the connection relationship between the controller 30, the first display device D3, and the second display device D3S. In the present embodiment, the first display device D3 and the second display device D3S are connected to the controller 30. The first display device D3, the second display device D3S, and the controller 30 may be connected via a communication network such as CAN.

[0041] The space recognition device S7 detects the presence or absence of objects existing in the space around the excavator 100, the distance to the objects, and the like. The space recognition device S7 outputs the result of measuring the space to the controller 30 as measurement information.

[0042] The first display device D3 includes a conversion processing unit D3a that generates an image. In the present embodiment, the conversion processing unit D3a generates a display camera image based on the output of a camera as the imaging device S6. The imaging device S6 is connected to the first display device D3 via, for example, a dedicated line.

[0043] The conversion processing unit D3a generates a display image based on the output of the controller 30. In the present embodiment, the conversion processing unit D3a converts various types of information output by the controller 30 into an image signal. The information output by the controller 30 includes, for example, data indicating the temperature of the engine cooling water, data indicating the temperature of the hydraulic oil, data indicating the remaining fuel amount, data indicating the remaining amount of urea water, data indicating the position of the working part of the bucket 6, data indicating the orientation of the slope surface of the work target, data indicating the orientation of the excavator 100, data indicating the operation direction for making the excavator 100 face the slope surface directly, and the like.

[0044] Similar to the first display device D3, the second display device D3S includes a conversion processing unit D3Sa that generates an image. In the present embodiment, the second display device D3S is not directly connected to the imaging device S6. Therefore, the conversion processing unit D3Sa does not generate a camera image. However, the conversion processing unit D3Sa may generate a camera image when the second display device D3S is directly connected to the imaging device S6.

[0045] The conversion processing unit D3Sa generates a display image based on the output of the controller 30. In the present embodiment, the conversion processing unit D3Sa converts various types of information output by the controller 30 into an image signal.

[0046] The conversion processing unit D3a may be implemented as a function of the controller 30 instead of a function of the first display device D3. The same applies to the conversion processing unit D3Sa. In this case, the imaging device S6 is connected to the controller 30 instead of the first display device D3.

[0047] The first display device D3 and the second display device D3S operate by receiving power supply from the battery 70. The battery 70 is charged with the power generated by the alternator 11a (generator) of the engine 11. The power of the battery 70 is supplied not only to the controller 30, the first display device D3, and the second display device D3S, but also to the electrical components 72 of the excavator 100 and the like. The starter 11b of the engine 11 is driven by the power from the battery 70 to start the engine 11.

[0048] The engine 11 is controlled by the engine control unit D7. Various data indicating the state of the engine 11 are constantly transmitted from the engine control unit D7 to the controller 30. The various data indicating the state of the engine 11 are an example of the operation information of the excavator 100, and include, for example, data indicating the coolant water temperature detected by the water temperature sensor 11c as the operation information acquisition unit. The controller 30 stores this data in a temporary storage unit (memory) 30a and can transmit it to the first display device D3 when necessary.

[0049] The following various data are supplied to the controller 30 as the operation information of the excavator 100 and stored in the temporary storage unit 30a of the controller 30.

[0050] For example, data indicating the swash plate tilt angle is supplied from the regulator 13 of the main pump 14, which is a variable displacement hydraulic pump, to the controller 30. Also, data indicating the discharge pressure of the main pump 14 is supplied from the discharge pressure sensor 14b to the controller 30. These data are stored in the temporary storage unit 30a. Further, an oil temperature sensor 14c is provided in the pipeline between the tank storing the hydraulic oil sucked by the main pump 14 and the main pump 14, and data representing the temperature of the hydraulic oil flowing through the pipeline is supplied from the oil temperature sensor 14c to the controller 30. The regulator 13, the discharge pressure sensor 14b, and the oil temperature sensor 14c are examples of the operation information acquisition unit.

[0051] Data indicating the fuel storage amount is supplied from the fuel storage amount detection unit 55a in the fuel storage unit 55 to the controller 30. In the present embodiment, data indicating the remaining amount state of the fuel is supplied from the fuel remaining amount sensor as the fuel storage amount detection unit 55a in the fuel tank as the fuel storage unit 55 to the controller 30.

[0052] Specifically, the fuel remaining amount sensor is composed of a float that follows the liquid level and a variable resistor (potentiometer) that converts the up-and-down movement amount of the float into a resistance value. With this configuration, the fuel remaining amount sensor can display the fuel remaining amount state continuously on the first display device D3. The detection method of the fuel storage capacity detection unit can be appropriately selected according to the usage environment, etc., and a detection method that can display the fuel remaining amount state step by step may be adopted. These configurations are the same for the urea water tank.

[0053] The operation device 26 is provided near the driver's seat in the cabin 10 and is used for the operator to operate various driven elements. Specifically, the operation device 26 is used for the operator to operate hydraulic actuators such as the left and right traveling hydraulic motors, boom cylinder 7, arm cylinder 8, bucket cylinder 9, and slewing hydraulic motor. As a result, the operator can operate the driven elements driven by the hydraulic actuators. The operation device 26 includes a pedal device and a lever device for operating each driven element.

[0054] The operation sensor 29 is configured to detect the operation content of the operator using the operation device 26. In this embodiment, the operation sensor 29 detects the operation direction and operation amount of the operation device 26 corresponding to each hydraulic actuator, and outputs an electrical signal (hereinafter also referred to as an operation signal) corresponding to the detected value to the controller 30. In this embodiment, the controller 30 controls the opening area of the proportional valve 31 according to the output of the operation sensor 29. Then, the controller 30 supplies the hydraulic oil discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17. The pressure (pilot pressure) of the hydraulic oil supplied to each pilot port is, in principle, the pressure corresponding to the operation direction and operation amount of the operation device 26 corresponding to each hydraulic actuator. In this way, the operation device 26 is configured to be able to supply the hydraulic oil discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17. Thereby, the hydraulic actuator can be driven.

[0055] Further, the direction switching valve that drives each hydraulic actuator built in the control valve unit 17 may be an electromagnetic solenoid type. In this case, the operation signal output from the operation device 26 may be directly input to the control valve unit 17 (i.e., to the electromagnetic solenoid type direction switching valve).

[0056] Note that the operation device 26 may be a hydraulic pilot type. Specifically, the operation device 26 utilizes the hydraulic oil supplied from the pilot pump 15 through the pilot line and outputs a pilot pressure corresponding to the operation content to the secondary side pilot line. And the secondary side pilot line is connected to the control valve unit 17. Thereby, a pilot pressure corresponding to the operation content regarding various driven elements (hydraulic actuators) in the operation device 26 can be input to the control valve unit 17. Therefore, the control valve unit 17 can drive each hydraulic actuator according to the operation content of the operation device 26 by an operator or the like. In this case, an operation sensor 29 capable of acquiring information regarding the operation state of the operation device 26 is provided, and the output of the operation sensor 29 is taken into the controller 30. Thereby, the controller 30 can grasp the operation state of the operation device 26. The operation sensor 29 is, for example, a pressure sensor that acquires information regarding the pilot pressure (operation pressure) of the secondary side pilot line of the operation device 26.

[0057] Also, part or all of the hydraulic actuators may be replaced with electric actuators. In this case, for example, the controller 30 may output an operation command corresponding to the operation content of the operation device 26 or the content of the remote operation defined by the remote operation signal to the electric actuator or a driver that drives the electric actuator. Further, by inputting an operation signal from the operation device 26 to the electric actuator or a driver or the like, the electric actuator may be configured to be operable by the operation device 26.

[0058] In addition, when the excavator 100 is solely remotely operated or solely operates by a fully automatic driving function, the operating device 26 may be omitted.

[0059] The proportional valve 31 functions as a control valve for machine control and is provided for each driven element (hydraulic actuator) to be operated by the operating device 26 and for each operating direction of the driven element (hydraulic actuator) (for example, the raising direction and the lowering direction of the boom 4). For example, two proportional valves 31 are provided for each double-acting hydraulic actuator for driving the lower traveling body 1, the upper swing body 3, the boom 4, the arm 5, the bucket 6, and the like. The proportional valve 31 is provided, for example, in a pilot line between the pilot pump 15 and the control valve unit 17, and may be configured to be able to change its flow passage area (that is, the cross-sectional area through which the hydraulic oil can flow). Thereby, the proportional valve 31 can output a predetermined pilot pressure to the secondary-side pilot line by using the hydraulic oil of the pilot pump 15 supplied through the primary-side pilot line. Therefore, the proportional valve 31 can apply a predetermined pilot pressure corresponding to the operation command from the controller 30 to the control valve unit 17. Thus, for example, the controller 30 can directly supply a pilot pressure corresponding to the operation content (operation signal) of the operating device 26 from the proportional valve 31 to the control valve unit 17, and realize the operation of the excavator 100 based on the operation of the operator.

[0060] In addition, the controller 30 may control the proportional valve 31 to realize the automatic driving function of the excavator 100. Specifically, the controller 30 outputs an operation command corresponding to the automatic driving function to the proportional valve 31 from the proportional valve 31. Thereby, the controller 30 can realize the operation of the excavator 100 by the automatic driving function.

[0061] Further, the controller 30 controls the proportional valve 31 to realize the remote operation of the excavator 100. Specifically, the controller 30 outputs an operation command corresponding to the content of the operation specified by the operation signal received from the remote operation room RC to the proportional valve 31 by the communication device T1. Thereby, the controller 30 can supply a pilot pressure corresponding to the content of the remote operation from the proportional valve 31 to the control valve unit 17, and realize the operation of the excavator 100 based on the remote operation of the operator.

[0062] In addition, when the operating device 26 is a hydraulic pilot type, a shuttle valve may be provided in the pilot line between the operating device 26 and the proportional valve 31 and the control valve unit 17. The shuttle valve has two inlet ports and one outlet port, and outputs the hydraulic oil having the higher pilot pressure among the pilot pressures input to the two inlet ports to the outlet port. The shuttle valve is provided for each driven element (hydraulic actuator) to be operated by the operating device 26 and for each operating direction of the driven element (hydraulic actuator), similar to the proportional valve 31. For example, two shuttle valves are provided for each double-acting hydraulic actuator for driving the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, the bucket 6, and the like. One of the two inlet ports of the shuttle valve is connected to the secondary pilot line of the operating device 26 (specifically, the above-described lever device or pedal device included in the operating device 26), and the other is connected to the secondary pilot line of the proportional valve 31. The outlet port of the shuttle valve is connected to the pilot port of the corresponding direction switching valve of the control valve unit 17 through the pilot line. The corresponding direction switching valve is a direction switching valve that drives the hydraulic actuator that is the operation target of the above-described lever device or pedal device connected to one inlet port of the shuttle valve. Therefore, each of these shuttle valves can apply the higher one of the pilot pressure of the secondary pilot line of the operating device 26 and the pilot pressure of the secondary pilot line of the proportional valve 31 to the pilot port of the corresponding direction switching valve. That is, the controller 30 can control the corresponding direction switching valve without depending on the operation of the operator on the operating device 26 by outputting a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26 from the proportional valve 31. Therefore, the controller 30 can control the operation of the driven elements (lower traveling body 1, upper slewing body 3, boom 4, arm 5, bucket 6) regardless of the operation state of the operator on the operating device 26, and can realize an automatic operation function and a remote operation function.

[0063] Also, when the operating device 26 is a hydraulic pilot type, in addition to the shuttle valve, a pressure reducing valve may be provided in the pilot line between the operating device 26 and the shuttle valve. The pressure reducing valve operates, for example, in response to a control signal input from the controller 30 and is configured to be able to change its flow passage area. Thereby, when the operating device 26 is being operated by the operator, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26. Further, the controller 30 can reduce the pilot pressure output from the operating device 26 with the pressure reducing valve and make it lower than the pilot pressure output from the proportional valve 31, even when the operating device 26 is being operated, for example. Therefore, by controlling the proportional valve 31 and the pressure reducing valve, the controller 30 can surely act a desired pilot pressure on the pilot port of the direction switching valve in the control valve unit 17 regardless of the operation content of the operating device 26, for example. Thus, the controller 30 can more appropriately realize the automatic operation function and the remote operation function of the excavator 100 by controlling the pressure reducing valve in addition to the proportional valve 31, for example.

[0064] As shown in FIG. 3, the communication system of the excavator 100 according to the present embodiment includes a communication device T1.

[0065] The communication device T1 is connected to an external communication line and communicates with a device provided separately from the excavator 100. The device provided separately from the excavator 100 may include, in addition to the devices outside the excavator 100, a portable terminal device (mobile terminal) brought into the cab 10 by the user of the excavator 100. The communication device T1 may include, for example, a mobile communication module compliant with standards such as 4G (4th Generation) and 5G (5th Generation). Further, the communication device T1 may include, for example, a satellite communication module. Further, the communication device T1 may include, for example, a Wi-Fi communication module, a Bluetooth (registered trademark) communication module, or the like. Also, when there are a plurality of connectable communication lines, the communication device T1 may include a plurality of communication devices T1 according to the type of the communication line.

[0066] For example, the communication device T1 communicates with an external device such as a remote operation room in the work site through a local communication line constructed in the work site. The local communication line is, for example, a mobile communication line based on local 5G (so-called local 5G) constructed in the work site or a local network based on Wi-Fi.

[0067] Further, the communication device T1 is configured to transmit and receive information to and from a communication device installed in a remote operation room through a wide-area communication line including the work site, that is, a wide-area network.

[0068] <Outline of processing performed by the controller> The controller 30 according to the present embodiment can also perform work with a fully automatic operation function by referring to the design data stored in the design data storage unit D4A (see FIG. 4) of the auxiliary storage device D4. Therefore, in the present embodiment, in order to grasp the situation of other work machines (for example, an excavator or a dump truck) around the excavator 100, as measurement information of the space recognition device S7 mounted on the excavator 100, three-dimensional point cloud data including the shape and distance of the objects existing around is acquired.

[0069] In order for the controller 30 to autonomously control the excavator 100 in the fully automatic operation function, it is necessary to recognize the position, orientation, and posture of other construction machines (for example, excavators, dump trucks, etc.) existing around the excavator 100. However, since construction machines are often composed of a plurality of parts including a drive mechanism and the like, it is often difficult to accurately recognize the position, orientation, and posture of the construction machine.

[0070] Therefore, the controller 30 according to the present embodiment identifies the shape of other construction machines existing around the excavator (an example of a construction machine) for each part constituting the other construction machine based on the measurement information represented by the measurement result of the space recognition device S7. Then, by identifying each part constituting the other construction machine, the controller 30 can recognize the current situation of the other construction machine and accurately grasp, for example, the position, orientation, and posture of the construction machine.

[0071] In the present embodiment, an example is given in which the controller 30 uses the image information captured by the imaging device S6 in order to identify the point cloud represented in the measurement information for each part of the other construction machine. However, the present embodiment shows an example of a method for identifying the point cloud represented in the measurement information for each part of the other construction machine, and other methods may be used.

[0072] Then, the controller 30 estimates at least one of the position, orientation, posture, movement, and work content of the other construction machine based on each of the parts identified as the other construction machine.

[0073] <Block configuration of the excavator controller> FIG. 4 is a functional block diagram showing a configuration example of the controller 30 of the excavator 100 according to the present embodiment. In the example shown in FIG. 4, the block configuration of the controller 30 of the excavator 100 is shown.

[0074] The controller 30 receives information output by the boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, machine body inclination sensor S4, slewing angle sensor S5, imaging device S6, space recognition device S7, input device D2, communication device T1, positioning device PS, etc. Then, based on the received information and the information stored in the auxiliary storage device D4, it executes various calculations and outputs the calculation results to the first display device D3, proportional valve 31, etc.

[0075] Note that in this embodiment, an example in which the controller 30 controls the excavator 100 will be described, but a part of the functions of the controller 30 may be realized by other controllers (control devices). That is, the functions of the controller 30 may be distributed and realized by a plurality of controllers mounted on the excavator 100.

[0076] For example, based on the inputs of the boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, machine body inclination sensor S4, and slewing angle sensor S5, the controller 30 can grasp (estimate) the position of the tip (bucket 6) of the attachment AT. Therefore, while grasping the position of the tip of the attachment AT, the controller 30 can control the operation of the excavator 100 by its automatic driving function.

[0077] The excavator 100 operates an actuator (for example, a hydraulic actuator) according to the operation of an operator boarding the cab 10, and drives operating elements (hereinafter, "driven elements") such as the lower traveling body 1, upper slewing body 3, boom 4, arm 5, and bucket 6.

[0078] Alternatively, or in addition to being configured to be operable by the operator in the cab 10, the excavator 100 may be configured to be remotely operable from outside the excavator 100. When the excavator 100 is remotely operated, the inside of the cab 10 may be unmanned.

[0079] Further, the excavator 100 may automatically operate the actuator regardless of the content of the operator's operation. Thereby, the excavator 100 realizes a function of automatically operating at least a part of the driven elements such as the lower traveling body 1, the upper swing body 3, the boom 4, the arm 5, and the bucket 6, that is, a so-called "automatic operation function" or "machine control function".

[0080] The automatic operation function may include a function of automatically operating driven elements (actuators) other than the driven element (actuator) of the operation target in response to an operation on the operator's operation device 26 or a remote operation, that is, a so-called "semiautomatic operation function" or "operation support type machine control function". Further, the automatic operation function may include a function of automatically operating at least a part of a plurality of driven elements (hydraulic actuators) on the premise that there is no operation on the operator's operation device 26 or a remote operation, that is, a so-called "fully automatic operation function" or "fully automatic type machine control function". In the excavator 100, when the fully automatic operation function is valid, the inside of the cab 10 may be unmanned. Also, the semiautomatic operation function, the fully automatic operation function, etc. may include a mode in which the operation content of the driven element (actuator) of the automatic operation target is automatically determined according to a rule defined in advance. Further, the semiautomatic operation function, the fully automatic operation function, etc. may include a mode (so-called "automatic operation function") in which the excavator 100 autonomously makes various determinations and the operation content of the driven element (hydraulic actuator) of the automatic operation target is determined autonomously according to the determination result.

[0081] Specifically, when the operator operates the arm 5 through the operating device 26, the controller 30 may automatically operate at least one of the boom 4 and the bucket 6 so that a preset target design surface (hereinafter simply referred to as the "design surface") coincides with the tip position of the bucket 6. Further, the controller 30 may also automatically operate the arm 5 regardless of the operating state of the operating device 26 that operates the arm 5. That is, the controller 30 may trigger an operation device 26 by an operator to cause an attachment to perform a preset operation. Hereinafter, a function of the controller 30 that operates at least one of the boom 4 and the bucket 6 in addition to the arm 5 according to an operation of the operating device 26 corresponding to the arm 5 is referred to as a "semiautomatic operation function". The semiautomatic operation function may be executed, for example, by operating a predetermined switch (hereinafter referred to as an "MC (Machine Control) switch") disposed at the tip of any one of the lever devices included in the operating device 26.

[0082] The auxiliary storage device D4 stores a design data storage unit D4A and a learned model LM.

[0083] The design data storage unit D4A stores design data. The design data includes construction data indicating a three-dimensional shape after the excavator 100 has performed construction at the work site. The construction data includes position data of a construction target in a world coordinate system indicated by GNSS and three-dimensional shape data after construction. For example, the design data includes position data and three-dimensional shape data of a target design surface formed after the excavator 100 has excavated earth and sand.

[0084] The position data is expressed, for example, in a reference coordinate system similar to the position data acquired by GNSS. The reference coordinate system is, for example, the World Geodetic System. The World Geodetic System is a three-dimensional orthogonal XYZ coordinate system with the origin at the center of gravity of the earth, the X-axis in the direction of the intersection of the Greenwich meridian and the equator, the Y-axis in the direction of 90 degrees east longitude, and the Z-axis in the direction of the North Pole.

[0085] When the image information captured by the imaging device S6 is input from the input layer, the learned model LM outputs classification image information indicating an image in which the machine tool represented in the image information is color-coded for each part, from the output layer. In this embodiment, an example of outputting the result of color-coding for each part will be described. However, this embodiment is not limited to the method of outputting the result of color-coding for each part, and any information that can classify each part may be used.

[0086] As the machine learning used for generating the learned model LM, for example, a neural network may be applied. Specifically, it may be machine learning using a deep neural network (DNN), and deep learning (deep neural network) may be applied. As deep learning, for example, a convolutional neural network, RNN (Recurrent Neural Networks), or LSTM (Long Short Term Memory) may be applied.

[0087] The learned model LM is generated by performing machine learning based on a teacher dataset generated in advance in an information processing device (not shown).

[0088] Specifically, the learned model LM is generated by machine learning based on the image information including the machine tool appearing in the teacher dataset and the classification image information in which the machine tool appearing in the image information is color-coded for each part.

[0089] Note that the learned model LM may be updated by additionally training a new teacher dataset on an existing learned model LM.

[0090] The controller 30 includes an acquisition unit 301, a classification unit 302, an identification unit 303, an estimation unit 304, and an automatic control unit 305.

[0091] The acquisition unit 301 acquires various information from various sensors. For example, the acquisition unit 301 acquires image information captured by the imaging devices S6 (front camera S6F, left camera S6L, right camera S6R, and rear camera S6B). For example, the acquisition unit 301 acquires measurement information indicating the results measured by the space recognition devices S7 (rear space recognition device S7B, left space recognition device S7L, right space recognition device S7R, and front space recognition device S7F).

[0092] The acquisition unit 301 acquires detection information detected by each of the boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, body tilt sensor S4, and swing angle sensor S5. The acquisition unit 301 acquires the position and orientation of the excavator 100 from the positioning device PS.

[0093] The acquisition unit 301 acquires design data corresponding to the work site where the excavator 100 is working from the design data storage unit D4A.

[0094] The classification unit 302 inputs the image information captured by the imaging device S6 acquired by the acquisition unit 301 into the input layer of the learned model LM, and receives, from the output layer of the learned model LM, classification image information in which the construction machines represented in the image information are color-coded for each part. In the present embodiment, each part of the construction machine is classified using the learned model LM. By using this method, the operator does not need to set specific conditions for classifying the parts for each construction machine, so the burden can be reduced.

[0095] FIG. 5 is a diagram illustrating image information captured by the imaging device S6 according to the present embodiment. In the image information 1500 shown in FIG. 5, a dump truck area 1501 and an excavator area 1502 are represented.

[0096] The classification unit 302 according to the present embodiment inputs the image information 1500 into the learned model LM, and receives classification image information in which the dump truck area 1501 and the excavator area 1502 represented in the image information are color-coded for each part.

[0097] FIG. 6 is a diagram illustrating the classified image information received by the classification unit 302 according to the present embodiment. In the classified image information 1600 shown in FIG. 6, the color separation results are shown for each part of the dump truck area 1501 and the excavator area 1502.

[0098] In the classified image information 1600, as a part shown in the dump truck area 1501, the area 1612 of the cargo bed is color-separated. Although not shown in FIG. 6, the area of the main body part (including the truck cabin, frame, wheels, etc.) of the dump truck area 1501 is also color-separated. The area 1612 of the cargo bed is painted, for example, in light blue, and the area of the main body part is painted, for example, in orange.

[0099] In the classified image information 1600, as parts shown in the excavator area 1502, the area 1621 of the lower traveling body, the area 1622 of the upper slewing body, the area 1623 of the boom, the area 1624 of the arm, and the area 1625 of the bucket are color-separated. The area 1621 of the lower traveling body is painted, for example, in loess color, the area 1622 of the upper slewing body is painted, for example, in red, the area 1623 of the boom is painted, for example, in blue, the area 1624 of the arm is painted, for example, in yellow, and the area 1625 of the bucket is painted, for example, in green.

[0100] In the classified image information of the present embodiment, each part is painted with the color associated with that part. Thus, it is possible to recognize which part of the working machine by the color shown in the classified image information.

[0101] Note that the classification shown in FIG. 6 is an example and is not intended to limit the classification method. For example, the main body part of the dump truck may be divided into a truck cabin, a frame, and wheels. Further, the cargo bed of the dump truck may be divided into a cargo bed and a tarpaulin. As another example, instead of dividing the excavator into the upper slewing body, the lower traveling body, the boom, the arm, and the bucket, it may be sufficient to divide at least two or more of the upper slewing body, the lower traveling body, the boom, the arm, and the bucket.

[0102] This embodiment does not limit the construction machine color-coded for each part by the learned model LM to an excavator and a dump truck, and it may be, for example, a bulldozer, a crane, or a wheel loader. For example, when color-coding a bulldozer, it is conceivable to paint the blade and the body in different colors.

[0103] Note that, in this embodiment, although a method using the learned model LM is described as a classification method for each part, the method is not limited to the method using the learned model LM. For example, the construction machine may be color-coded in advance for each part, and the controller 30 may classify the parts of the construction machine based on the difference in colors shown in the image information. Furthermore, markers may be attached in advance to the construction machine for each part, and the controller 30 may classify the parts of the construction machine based on the markers shown in the image information.

[0104] Returning to FIG. 4, the identification unit 303 identifies, for each part of the construction machine, a point cloud in which the shape of another construction machine is represented by the measurement information, based on the classified image information (the result of color-coding each part of the construction machine) output from the classification unit 302.

[0105] In this embodiment, calibration has been performed in advance regarding the positional relationship between the spatial recognition device S7 and the imaging device S6. Therefore, the correspondence between the area imaged by the imaging device S6 and a part of the point cloud included in the measurement information can be recognized. Accordingly, the identification unit 303 of this embodiment can cluster (identify) the point cloud corresponding to each part from the measurement information based on the parts of the construction machine classified by the image information.

[0106] In addition, the identification unit 303 assigns the colors included in the image information to each point of the point cloud included in the measurement information. The assignment of the colors can be realized from the correspondence between the area imaged by the imaging device S6 and a part of the point cloud measured by the spatial recognition device S7.

[0107] Furthermore, for each part of the working machine, the identification unit 303 fits (assigns) the point cloud clustered (identified) as that part with a rectangular parallelepiped. In this embodiment, an example of fitting with a rectangular parallelepiped will be described, but the three-dimensional shape to be fitted to the point cloud is not limited to a rectangular parallelepiped. An appropriate three-dimensional shape may be used according to the embodiment.

[0108] FIG. 7 is a diagram showing in a virtual three-dimensional space the result of the identification unit 303 according to this embodiment fitting the point cloud included in the measurement information with a rectangular parallelepiped for each part of the working machine.

[0109] The virtual three-dimensional space shown in FIG. 7 is a coordinate system of a virtual three-dimensional space based on the excavator 100 in order to represent the measurement result of the space recognition device S7. For example, with the center of the excavator 100 as the origin, the point cloud of the object detected by the space recognition device S7 is represented. For example, in the virtual three-dimensional space 1700, a region 1701 where the dump truck 1800 (see FIG. 8) exists and a region 1702 where another excavator 100A (see FIG. 8) exists are represented.

[0110] Furthermore, the virtual three-dimensional space 1700 shown in FIG. 7 represents a state in which, for each point included in the point cloud included in the measurement information, the color of the image information corresponding to the point is assigned to the position indicated by the direction and distance of the point.

[0111] Furthermore, rectangular parallelepipeds 1711 to 1717 fitted for each part of the working machine (dump truck 1800 and excavator 100A) are represented in the virtual three-dimensional space. Further, each line of the rectangular parallelepipeds 1711 to 1717 may be color-coded according to the part, or may be represented by the color represented by the classification image information, for example.

[0112] For example, the rectangular parallelepiped 1711 indicates the region identified as the main body of the dump truck 1800. The color of the line of the rectangular parallelepiped 1711 is, for example, orange. The rectangular parallelepiped 1712 indicates the region identified as the cargo bed of the dump truck. The color of the line of the rectangular parallelepiped 1712 is, for example, light blue.

[0113] As another example, the rectangular parallelepiped 1713 indicates a region identified as the lower traveling body of the excavator. The color of the lines of the rectangular parallelepiped 1713 is, for example, loess color. The rectangular parallelepiped 1714 indicates a region identified as the upper revolving body of the excavator. The color of the lines of the rectangular parallelepiped 1714 is, for example, red. The rectangular parallelepiped 1715 indicates a region identified as the boom of the excavator. The color of the lines of the rectangular parallelepiped 1715 is, for example, blue. The rectangular parallelepiped 1716 indicates a region identified as the arm of the excavator. The color of the lines of the rectangular parallelepiped 1716 is, for example, yellow. The rectangular parallelepiped 1717 indicates a region identified as the bucket of the excavator. The color of the lines of the rectangular parallelepiped 1717 is, for example, green.

[0114] Returning to FIG. 4, the estimation unit 304 estimates the position, orientation, posture, movement, and work content of the working machine based on the positions of the rectangular parallelepipeds fitted (assigned) to each part of the working machine. Note that in this embodiment, an example of estimating the position, orientation, posture, movement, and work content will be described, but the method is not limited to estimating all of the position, orientation, posture, movement, and work content, and any one or more of the position, orientation, posture, movement, and work content may be estimated.

[0115] FIG. 8 is a top view showing the state in which the rectangular parallelepipeds are fitted to each part of the working machine by the identification unit 303 according to this embodiment. In the example shown in FIG. 8, the excavator 100A and the dump truck 1800 are detected.

[0116] In the example shown in FIG. 8, the excavator 100A is fitted with the rectangular parallelepipeds 1713 to 1717 for each part, and the dump truck 1800 is fitted with the rectangular parallelepipeds 1711 and 1712 for each part. Then, the estimation unit 304 estimates any one or more of the position, orientation, posture, movement, and work content based on the rectangular parallelepipeds fitted to each part.

[0117] For example, the estimation unit 304 estimates the position where the center of gravity 1801 of the rectangular parallelepiped 1713 corresponding to the lower traveling body 1 of the excavator 100A is projected onto a horizontal plane (e.g., the ground) as the position of the excavator 100A.

[0118] The estimation of the orientation of the excavator 100A is performed for each of the upper slewing body 3 and the lower traveling body 1. For example, the estimation unit 304 estimates the direction parallel to the longitudinal side (L2) among the two sides (L1, L2) other than the height direction of the rectangular parallelepiped 1713 corresponding to the lower traveling body 1 of the excavator 100A as the travelable direction 1802 of the lower traveling body 1, and estimates the direction substantially parallel to the short side direction as the width direction of the lower traveling body 1. Thereby, the estimation unit 304 can estimate the travelable direction 1802 as the orientation of the lower traveling body 1.

[0119] Furthermore, the estimation unit 304 estimates the direction from the position where the center of gravity 1725 of the rectangular parallelepiped 1715 corresponding to the boom 4 is projected onto the horizontal plane to the position where the center of gravity 1726 of the rectangular parallelepiped 1716 corresponding to the arm 5 is projected onto the horizontal plane as the forward direction (orientation) 1803 of the upper slewing body 3. Note that in this embodiment, the estimation unit 304 is not limited to this estimation method. For example, based on the position where the center of gravity of the rectangular parallelepiped 1717 corresponding to the bucket 6 is projected onto the horizontal plane and the position where the center of gravity 1725 of the rectangular parallelepiped 1715 corresponding to the boom 4 or the center of gravity 1726 of the rectangular parallelepiped 1716 corresponding to the arm 5 is projected onto the horizontal plane, the front-rear direction of the upper slewing body 3 may be estimated. Furthermore, the estimation unit 304 may estimate the front-rear direction of the upper slewing body 3 based on the positional relationship between the rectangular parallelepiped 1717 corresponding to the upper slewing body 3 and the rectangular parallelepipeds 1715, 1716, 1717 corresponding to each of the attachments AT.

[0120] In this embodiment, an example of estimating the orientation of the upper slewing body 3 based on the centers of gravity of the rectangular parallelepipeds 1715 and 1716 will be described. However, the method of estimating the orientation based on the center of gravity is not limited, and any method of estimating the orientation based on the positional relationship of the estimated parts may be used. For example, if the positional relationship between the part identified as the boom 4 and the part identified as the arm 5 can be recognized, the orientation of the upper slewing body 3 can be specified.

[0121] Furthermore, the estimation unit 304 estimates the posture of the excavator 100A based on the positional relationship of the rectangular parallelepipeds 1713 to 1717 fitted to each part of the excavator 100A. For example, when the rectangular parallelepipeds 1716 and 1717 corresponding to the arm 5 and the bucket 6 exist downward of the rectangular parallelepiped 1715 corresponding to the boom 4, the estimation unit 304 estimates that the excavator 100A is in a traveling posture with the arm 5 and the bucket 6 folded under the boom 4.

[0122] In addition, the estimation unit 304 estimates the operation of the excavator 100A based on changes in the positions of at least one or more of the rectangular parallelepipeds 1715 to 1717. For example, when the rectangular parallelepiped 1714 corresponding to the upper swing body 3 rotates compared to the rectangular parallelepiped 1713 corresponding to the lower traveling body 1, the estimation unit 304 estimates that the operation of the excavator 100A is a swing. As another example, when the rectangular parallelepiped 1715 corresponding to the boom 4 rotates upward, the estimation unit 304 estimates that the operation of the excavator 100A is a boom lift.

[0123] Furthermore, the estimation unit 304 estimates the work content of the excavator 100A based on changes in the positions of at least one or more of the rectangular parallelepipeds 1715 to 1717. For example, when at least a part of the rectangular parallelepiped 1717 corresponding to the bucket 6 is located downward from the ground, the estimation unit 304 estimates that the work of the excavator 100A is excavation. Also, when the rectangular parallelepiped 1717 corresponding to the bucket 6 moves upward, the estimation unit 304 estimates that it is earth and sand transportation. Further, when the rectangular parallelepiped 1717 corresponding to the bucket 6 tilts on the dump truck 1800, the estimation unit 304 estimates that it is dumping.

[0124] In addition, in the dump truck 1800, the estimation unit 304 estimates the position of the dump truck 1800 as the position where the center of gravity 1721 of the rectangular parallelepiped corresponding to the main body is projected onto the horizontal plane (ground).

[0125] Furthermore, the estimation unit 304 estimates the direction 1811 from the position where the center of gravity 1722 of the rectangular parallelepiped 1712 corresponding to the cargo bed is projected onto the horizontal plane (ground) to the position where the center of gravity 1721 of the rectangular parallelepiped 1711 corresponding to the main body is projected onto the horizontal plane as the forward direction (orientation) of the dump truck 1800.

[0126] Furthermore, the estimation unit 304 estimates the attitude of the dump truck 1800 based on the positional relationship between the rectangular parallelepiped 1711 corresponding to the main body and the rectangular parallelepiped 1712 corresponding to the cargo bed.

[0127] As an estimation of an attitude different from that of the present embodiment, there is an estimation of the tilt angle of the cargo bed of the dump truck. For this estimation, among the cargo bed of the dump truck, the point cloud of the cargo bed and the point cloud of the tilt are each fitted to different rectangular parallelepipeds. Then, the estimation unit 304 estimates the tilt angle of the cargo bed of the dump truck 1800 based on the positional relationship between the rectangular parallelepiped corresponding to the cargo bed and the rectangular parallelepiped corresponding to the tilt.

[0128] FIG. 9 is a diagram illustrating the correspondence between the positional relationship of the rectangular parallelepipeds fitted to each part of the dump truck 1800 and the attitude of the dump truck 1800.

[0129] In FIG. 9(A), the rectangular parallelepiped 1712 corresponding to the cargo bed is tilted by an angle θ compared to the rectangular parallelepiped 1711 corresponding to the main body. Since the rectangular parallelepiped 1712 corresponding to the cargo bed is tilted by an angle θ compared to the rectangular parallelepiped 1711 corresponding to the main body, the estimation unit 304 estimates that the dump truck 1800 is in the process of dumping up.

[0130] In FIG. 9(B), the rectangular parallelepiped 1712 corresponding to the cargo bed and the rectangular parallelepiped 1711 corresponding to the main body are arranged substantially in parallel. When the rectangular parallelepiped 1712 corresponding to the cargo bed and the rectangular parallelepiped 1711 corresponding to the main body are substantially parallel and the rectangular parallelepiped 1711 has not moved (the position has not changed substantially between the previous detection result and the current detection result), the estimation unit 304 estimates that the dump truck 1800 is waiting to load earth and sand or the like on the cargo bed.

[0131] In this way, the estimation unit 304 estimates whether the dump is raised or not according to whether the angle between the rectangular parallelepiped 1711 corresponding to the main body and the rectangular parallelepiped 1712 corresponding to the loading platform is inclined by a predetermined angle or more (let the angle θ ≥ the predetermined angle).

[0132] Furthermore, the estimation unit 304 estimates the moving direction of the working machine based on the position of the rectangular parallelepiped detected last time and the position of the rectangular parallelepiped detected this time. For example, the estimation unit 304 estimates the moving direction of the excavator 100A from the change in the position of the rectangular parallelepiped 1713 corresponding to the lower traveling body 1. Furthermore, the estimation unit 304 estimates whether the dump truck 1800 is traveling or not based on the change in the position of the rectangular parallelepiped 1711 corresponding to the main body of the dump truck 1800. Furthermore, the estimation unit estimates the speed of the working machine (for example, the excavator 100A or the dump truck 1800) from the amount of change in position.

[0133] The estimation unit 304 according to the present embodiment estimates the position, orientation, posture, movement, and work content of the working machine based on the result of identifying other working machines for each part. When the posture of another working machine changes, the controller 30 according to the present embodiment identifies the other working machine for each part and estimates the posture of the other working machine according to the identification result, so that the change in posture can be recognized. Therefore, an improvement in the recognition accuracy of the posture can be realized.

[0134] The present embodiment shows an example of an estimation method for the position, orientation, posture, movement, and work content of a working machine by the estimation unit 304, and is not limited to the said estimation method, and other estimation methods may be used. The estimation unit 304 according to the present embodiment can recognize the specific situation of the other working machine by estimating the position, orientation, posture, movement, and work content of the other working machine. Therefore, the excavator 100 can be operated in consideration of the specific situation of the other working machine, so that an improvement in safety and an improvement in work efficiency can be realized.

[0135] Further, the period at which the estimation unit 304 performs estimation may be determined according to the embodiment. For example, it may be the period at which the space recognition device S7 detects objects around the excavator 100. Furthermore, the period for performing estimation may be switched according to the application. For example, when performing peripheral monitoring, it may be once per second, and when performing safety confirmation around the excavator 100, it may be once every 0.1 seconds.

[0136] Returning to FIG. 4, the automatic control unit 305 is configured to be able to autonomously operate the excavator 100 based on the design data stored in the design data storage unit D4A and the estimation result by the estimation unit 304.

[0137] For example, the automatic control unit 305 operates the attachment AT so as to form the target design surface shown in the design data.

[0138] Then, when it is estimated by the estimation unit 304 that the dump truck 1800 is waiting to load earth and sand or the like on the loading platform, the automatic control unit 305 controls to discharge the earth and sand loaded in the bucket 6 onto the loading platform of the dump truck 1800.

[0139] Furthermore, when it is estimated that another excavator 100A is moving, the automatic control unit 305 autonomously operates the attachment AT and the lower traveling body 1 so as not to contact the other excavator 100A. As a method of autonomous operation, a well-known method may be used. For example, there is a method of generating an orbit in advance for the excavator 100 to move and operating according to the orbit.

[0140] The automatic control unit 305 according to the present embodiment generates an orbit in consideration of the moving direction, speed, and operation of other construction machines, so that contact with other construction machines can be suppressed and safety can be improved. Furthermore, since the automatic control unit 305 generates an orbit in consideration of the operation of other construction machines, cooperative work can be easily realized and work efficiency can be improved.

[0141] For example, the automatic control unit 305 generates a trajectory for operating the attachment AT or the lower traveling body 1, and outputs a control command (current command) to the proportional valve 31 so as to operate the attachment AT or the lower traveling body 1 according to the trajectory. The proportional valve 31 autonomously operates each actuator by individually adjusting the pilot pressure acting on the control valve corresponding to each actuator according to the control command. Note that the method of generating the trajectory is well-known, and thus the description thereof is omitted.

[0142] Next, the processing procedure executed by the controller 30 according to the present embodiment will be described. FIG. 10 is a flowchart showing the processing procedure until the controller 30 according to the present embodiment performs automatic control.

[0143] First, the acquisition unit 301 acquires the image information captured by the imaging device S6 and the measurement information measured by the space recognition device S7 (S2001).

[0144] The classification unit 302 inputs the image information into the learned model LM, and receives classification image information obtained by classifying the construction machine represented in the image information for each part from the learned model LM (S2002).

[0145] Based on the classification image information (classification result of the parts of the construction machine) received from the classification unit 302, the identification unit 303 identifies the point cloud included in the measurement information for each part of the construction machine (S2003).

[0146] The identification unit 303 fits the point cloud extracted for each part of the construction machine with a rectangular parallelepiped (S2004).

[0147] Based on the positions of the rectangular parallelepipeds fitted to each part of the construction machine, the estimation unit 304 estimates the position, orientation, posture, movement, and work content of the construction machine (S2005).

[0148] The automatic control unit 305 autonomously operates the excavator 100 based on the design data stored in the design data storage unit D4A and the estimation result by the estimation unit 304 (S2006).

[0149] In the present embodiment, by having the controller 30 with the above-described configuration, the situation of other construction machines (for example, dump trucks and other excavators) existing around the excavator 100 can be recognized, enabling automatic control considering such a situation.

[0150] In the present embodiment, the case where the controller 30 performs the above-described processing has been described. However, the present embodiment does not limit the method by which the controller 30 performs the above-described processing. For example, the controller 30 may transmit the image information and the measurement information to an external server via the communication device T1. Then, the external server may perform the above-described processing and transmit the processing result to the communication device T1. And the controller 30 may autonomously operate the excavator 100 based on the received processing result.

[0151] (Modification example) The above-described embodiment has been described with an example of referring to the result of classifying the construction machine shown in the image information captured by the imaging device S6 for each part of the construction machine in order to identify the point cloud included in the measurement information for each part of the construction machine. However, the above-described embodiment does not limit the method of identifying the point cloud included in the measurement information for each part of the construction machine based on the result of classifying the construction machine shown in the image information for each part.

[0152] Therefore, in the modification example, a method will be described in which the controller 30 identifies the point cloud included in the measurement information for each of a plurality of parts constituting the construction machine without using the image information captured by the imaging device S6. In this modification example, the processing of the classification unit 302 is different from that of the above-described embodiment. Also, in this modification example, the identification unit 303 may be omitted.

[0153] The auxiliary storage device D4 according to this modification stores a learned model LM1 instead of the learned model LM of the above-described embodiment.

[0154] When the measurement information measured by the space recognition device S7 is input from the input layer, the learned model LM1 outputs, from the output layer, information indicating the result of identifying the point group of the shape of the machine tool represented by the measurement information for each part.

[0155] As the machine learning used for generating the learned model LM1, for example, a neural network may be applied, specifically, machine learning using a deep neural network (DNN), and deep learning (deep neural network) may be applied. As deep learning, for example, a convolutional neural network, RNN (Recurrent Neural Networks), or LSTM (Long Short Term Memory) may be applied.

[0156] The learned model LM1 is generated by performing machine learning based on a teacher dataset generated in advance in an information processing device (not shown).

[0157] Specifically, the learned model LM1 is generated by machine learning based on the measurement information detecting the machine tool included in the teacher dataset and the information identifying the machine tool detected by the measurement information for each part.

[0158] Then, the classification unit 302 inputs the measurement information measured by the space recognition device S7 acquired by the acquisition unit 301 to the input layer of the learned model LM1, and receives, from the output layer of the learned model LM1, information indicating the result of identifying the point group for each part of the machine tool detected by the measurement information. Then, the classification unit 302 can identify the point group corresponding to each part of the machine tool from the received information. Regarding other processes, the description is omitted as the same as in the above-described embodiment.

[0159] The controller 30 according to this modification example can identify a point group corresponding to each part of the working machine from the measurement information measured by the space recognition device S7 without using the image information captured by the imaging device S6. Therefore, it has the same effects as the above-described embodiment and can achieve cost reduction.

[0160] (Second Embodiment) In the above-described embodiment and modification example, the case where the controller 30 of the excavator 100 recognizes the surrounding situation and performs automatic control by identifying the parts constituting other working machines has been described. However, the above-described embodiment and modification example do not limit the purpose of identifying the parts constituting other working machines to performing automatic control. Therefore, in this embodiment, the case of monitoring the periphery of the excavator 100 will be described.

[0161] FIG. 11 is a schematic diagram showing a configuration example of the support system SYS of the excavator 100 according to this embodiment. FIG. 11 shows an example in which the excavator 100, the fixed-point measurement device 2150, the information center 2100, and the remote operation room RC are connected.

[0162] The excavator 100 uses the communication device T1 provided in the excavator 100 to transmit the detection results from various sensors provided in the excavator 100 to the information center 2100. For example, the excavator 100 transmits the image information captured by the imaging device S6 and the measurement information measured by the space recognition device S7 to the information center 2100.

[0163] The information center 2100 according to this embodiment has the same configuration as the controller 30 of the above-described embodiment. Therefore, the information center 2100 identifies the parts constituting other working machines based on the image information captured by the imaging device S6 and the measurement information measured by the space recognition device S7, and fits a rectangular parallelepiped to each identified part. Then, the information center 2100 fits a rectangular parallelepiped to each part and displays the information in which the color of the image information is assigned to each point of the point group on the display device 2110. Thereby, for example, the information as shown in FIG. 7 is displayed on the display device 2110.

[0164] The information displayed on the display device 2110 is not limited to the information as shown in FIG. 7. For example, the display device 2110 may display any one or more of the position, orientation, posture, movement, and work content of other construction machines estimated by the information center 2100. For example, the administrator of the information center 2100 can recognize the current situation of each construction machine at the work site by the display of the posture, movement, and work content of each construction machine.

[0165] Also, the above-described embodiment is not limited to the method of providing the imaging device S6 and the space recognition device S7 to the excavator 100. For example, it may be provided at the work site.

[0166] As a specific example, the fixed-point measurement device 2150 is provided at the work site where the excavator 100 performs work. The fixed-point measurement device 2150 has functions as the imaging device S6 and the space recognition device S7 described in the above-described embodiment.

[0167] The fixed-point measurement device 2150 transmits the image information obtained by imaging the work site and the measurement information obtained by measuring the work site to the information center 2100. Thereby, the information center 2100 can monitor the current situation for each part constituting the construction machine existing at the work site. Furthermore, the information center 2100 displays any one or more of the position, orientation, posture, movement, and work content of each construction machine estimated based on the identified part of the construction machine on the display device 2110.

[0168] Furthermore, the information center 2100 may combine and process the image information and measurement information received from the excavator 100 and the image information and measurement information received from the fixed-point measurement device 2150. The information center 2100 arranges point groups and colors based on each of the plurality of image information and measurement information in a virtual three-dimensional space. Other processes are the same as those in the above-described embodiment, and the description thereof is omitted.

[0169] Since the manager at the work site who is present in the information center 2100 can grasp the current status of each working machine at the work site, the progress of the construction can be managed and the work can be carried out according to the plan.

[0170] In the support system SYS according to the present embodiment, a remote operation room RC is provided. In the remote operation room RC, a display device DR, an operation device 26R, an operation sensor 29R, an operation seat DS, a remote controller R30, and a communication device T2 are provided.

[0171] The remote controller R30 may have the same functions as the controller 30 of the above-described embodiment. In this case, the remote controller R30 receives image information and measurement information via the information center 2100, and performs the same processing as the controller 30 to display the current status of each part constituting the working machine (which may include the excavator 100) present at the work site on the display device DR. Thereby, even when the operator OP is present in the remote operation room RC, the operator OP can confirm the situation around the excavator 100.

[0172] The operator OP present at the operation seat DS in the remote operation room RC confirms the surrounding situation and operates the operation device 26R as necessary. Then, the operation sensor 29R detects the operation content received by the operation device 26R.

[0173] Then, the remote controller R30 generates a control signal for operating the excavator 100 based on the detected operation content according to the detected operation content. Then, the communication device T2 transmits the generated control signal to the excavator 100. By transmitting the control signal, the remote controller R30 can control the excavator 100.

[0174] In this embodiment, the excavator 100 may perform automatic control in the same manner as in the first embodiment. Then, when the operator OP present in the remote operation room RC monitors the current situation at the work site and determines that it is better to operate by himself / herself, the remote controller R30 may switch the control so as to operate the excavator 100 with the operation device 26R according to the operation from the operator OP.

[0175] In this embodiment, an example has been described in which the identification result of the part constituting the work machine is displayed on the display device 2110 or the display device DR connected to the information center 2100. However, this embodiment does not limit the display destination to the outside of the excavator 100 such as the display device 2110 or the display device DR.

[0176] For example, the identification result of the part constituting the work machine may be displayed on the first display device D3 in the cabin 10. Further, the estimation results of the position, orientation, posture, movement, and work content of each work machine may be displayed on the first display device D3 in the cabin 10. In this case, the operator riding in the cabin 10 can recognize the surrounding situation from the display. Then, the operator can operate the excavator 100 in consideration of the surrounding situation.

[0177] <Function> In the above-described embodiments and modifications, the controller 30, the information center 2100, or the remote controller R30 performs the above-described control to identify each of a plurality of parts constituting another work machine, so that the current situation of the other work machine can be recognized. Therefore, since the work machine can be operated in consideration of the current situation of the other work machine, an improvement in safety can be realized.

[0178] As described above, the embodiments of the system for a work machine and the excavator according to the present invention have been described, but the present invention is not limited to the above embodiments. Various changes, modifications, substitutions, additions, deletions, and combinations are possible within the scope described in the claims. Naturally, they also belong to the technical scope of the present invention.

Explanation of Reference Numerals

[0179] 100 Excavator 1 Lower Travel Body 2 Swing Mechanism 3 Upper Swing Structure 4 Boom 5 Arm 6 Bucket S1 Boom Angle Sensor S2 Arm Angle Sensor S3 Bucket Angle Sensor S4 Machine Inclination Sensor S5 Swing Angle Sensor S6 Imaging Device S7 Spatial Recognition Device PS Positioning Device T1 Communication Device D4 Auxiliary Memory Device D4A Design Data Storage Unit LM, LM1 Learned Model 30 Controller 301 Acquisition Unit 302 Classification Unit 303 Identification Unit 304 Estimation Unit 305 Automatic Control Unit RC Remote Operation Cab R30 Remote Controller T2 Communication Device 2100 Information Center 2110 Display Device 2150 Fixed-Point Measurement Device

Claims

1. A space recognition device capable of measuring the shape of an object existing around the work machine, and a control device configured to identify the shape of another work machine existing around the work machine for each part constituting the other work machine based on measurement information representing the measurement result by the space recognition device. A system for a work machine comprising the above.

2. Based on each of the identified parts, the control device estimates at least one of the position, orientation, posture, movement, and work content of the other work machine. The system for a work machine according to Claim 1.

3. Based on the positional relationship of each of the parts, the control device estimates the orientation of the other work machine. The system for a work machine according to Claim 2.

4. The control device assigns a predetermined three-dimensional shape to each of the identified parts, and estimates at least one of the position, orientation, posture, movement, and work content of the other work machine based on the position of the three-dimensional shape assigned to each of the parts. The system for a work machine according to Claim 2.

5. The system further includes an imaging device capable of imaging the periphery of the work machine, and the control device has a learned model subjected to machine learning processing using teacher data so as to output a result of dividing each of the parts of the other work machine appearing in the image information when the image information imaged by the imaging device is input, and the control device identifies the shape of the other work machine appearing in the measurement information for each part constituting the other work machine based on the result of dividing the other work machine appearing in the image information output from the learned model into each of the parts. The system for a work machine according to Claim 1.

6. The control device has a learned model subjected to machine learning processing using teacher data so as to output a result of identifying the shape of the other work machine appearing in the measurement information for each of the parts when the measurement information is input, The system for a work machine according to Claim 1.

7. The control device identifies at least two or more of an upper swing body, a lower traveling body, a boom, an arm, and a bucket from the excavator appearing in the measurement information, or identifies at least two or more of a vehicle body, a truck cabin, wheels, and a loading platform from the dump truck appearing in the measurement information. The system for a working machine according to claim 1.

8. When the control device identifies the excavator, based on changes in the positions of at least one or more of the upper slewing body, the lower traveling body, the boom, the arm, and the bucket, it estimates the operation or work content of the excavator. The system for a working machine according to claim 7.

9. When the control device identifies the dump truck, based on the positional relationships of at least two or more of the vehicle body, the truck cabin, the wheels, and the loading platform, it estimates the attitude of the dump truck. The system for a working machine according to claim 7.

10. The space recognition device is provided on the working machine. The system for a working machine according to claim 1.

11. A space recognition device capable of measuring the shape of an object existing around the excavator, and A control device configured to identify the shape of another working machine existing around the excavator for each part constituting the other working machine based on the measurement information represented by the measurement result by the space recognition device. An excavator comprising the above.

Images