Image display system for work machines

The image display system for work machines addresses the issue of poor perspective in remote operations by generating three-dimensional images, enhancing efficiency and accuracy through improved distance perception.

JP7593599B2Active Publication Date: 2024-12-03KOMATSU LTD +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2023127017
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-08-03
Publication Date
2024-12-03
Estimated Expiration
2036-09-02

AI Technical Summary

Technical Problem

The use of two-dimensional images in remote operation of work machines, such as hydraulic excavators, leads to a poor sense of perspective, making it difficult to gauge distances between the work target and the machine, thereby reducing work efficiency.

Method used

An image display system that generates a three-dimensional image of the work target relative to the work tool, incorporating distance and positional information to provide a clearer perspective, using a position detection section, distance detection device, and processing device to create images on a display device.

Benefits of technology

The system enhances work efficiency by providing operators with a clear sense of distance and perspective, reducing errors and improving accuracy in operations involving revolving bodies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007593599000001
    Figure 0007593599000001
  • Figure 0007593599000002
    Figure 0007593599000002
  • Figure 0007593599000003
    Figure 0007593599000003
Patent Text Reader

Abstract

To suppress reduction in work efficiency in working using a work machine comprising a work apparatus having a work tool.SOLUTION: A work machine image display system to be applied to a work machine comprising a work apparatus having a work tool and a revolving superstructure to which the work apparatus is attached includes: a position detection unit for detecting at least one of the work apparatus's posture and position; a distance detection device for calculating information on a distance from the work machine to a work target; and a processing device for creating a first image using information on the work tool's position acquired by the position detection unit and the information on the distance calculated by the distance detection device to make the first image be displayed on a display device, the first image being an image of part on the work target facing the work tool, the part extending in a revolving direction of the revolving superstructure and including part corresponding to the work tool's portion.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] The present invention relates to an image display system for a work machine. [Background technology]

[0002] As described in Patent Document 1, a technique for remotely controlling a work machine such as a hydraulic excavator is known. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-294067 Summary of the Invention [Problem to be solved by the invention]

[0004] When remotely operating a work machine, if the image displayed is from the operator's viewpoint, the sense of perspective is poor because the displayed image is two-dimensional. This makes it difficult to grasp the distance between the work object and the work machine, which can lead to reduced work efficiency. Also, when an operator on board the work machine operates the work machine, depending on the operator's level of skill, it can be difficult to grasp the distance between the work machine and the work object, which can lead to reduced work efficiency.

[0005] An aspect of the present invention aims to suppress a decrease in work efficiency when working using a work machine equipped with a work implement having a work tool. [Means for solving the problem]

[0006] According to a first aspect, there is provided an image display system for a work machine that is applicable to a work machine that has a work implement and a rotating body to which the work implement is attached, and that includes: a position detection unit that detects the position of the work implement; a distance detection device that obtains information on the distance from the work machine to a work object; and a processing device that generates a first image using information on the position of the work implement obtained by the position detection unit and information on the position of the work object obtained from the distance information obtained by the distance detection device, and displays the first image on a display device, wherein the first image is generated based on information on a first position corresponding to the work implement on the surface of the work object opposite the work implement, and information on the position of a portion extending from the first position along the direction in which the rotating body rotates.

[0007] According to a second aspect, there is provided an image display system for a work machine in the first aspect, wherein the first image is an image of an arc whose radius is the distance between the rotation center axis of the rotating body and a part of the work implement, and whose center is a position corresponding to the rotation center axis.

[0008] According to a third aspect, there is provided an image display system for a work machine in the second aspect, wherein the work implement is a bucket and part of the work implement is the cutting edge of the bucket located in the center of the bucket in the width direction.

[0009] According to a fourth aspect, in any one of the first to third aspects, an image display system for a work machine is provided in which the processing device uses information on the position of the work object to generate a line image along the surface of the work object, combines it with the image of the work object, and displays it on the display device.

[0010] According to the fifth aspect, in the fourth aspect, an image display system for a work machine is provided, in which the line image comprises a plurality of first line images extending radially from a position corresponding to the central axis of rotation of the rotating body, and a plurality of second line images extending along the rotation direction of the rotating body with the central axis of rotation as the center.

[0011] According to a sixth aspect, in any one of the first to fifth aspects, there is provided an image display system for a work machine having an imaging device attached to the rotating body, and the processing device synthesizes the first image with a second image, which is an image of the work object captured by the imaging device, and displays the synthesized image on the display device.

[0012] According to the seventh aspect, in the sixth aspect, an image display system for a work machine is provided in which the processing device uses the attitude of the work machine to determine the area occupied by the work machine in the second image, and removes the obtained area from information on the shape of the work object.

[0013] According to the eighth aspect, there is provided an image display system for a work machine in which, in the sixth or seventh aspect, the imaging device, the position detection unit and the distance detection device are provided on the work machine, and the processing device and the display device are installed in a facility equipped with an operating device for remotely operating the work machine.

[0014] According to a ninth aspect, there is provided an image display system for a work machine that is applicable to a work machine that includes a work implement having a work tool, a rotating body to which the work implement is attached, a position detection unit that detects the position of the work implement, a distance detection device, and an imaging device, and the system includes a display device, and a processing device that generates a first image using information on the position of the work implement obtained by the position detection unit and information on the position of the work object obtained from information on the distance from the work machine to the work object determined by the distance detection device, and combines this with a second image, which is an image of the work object captured by the imaging device, and displays the result on the display device, wherein the first image is generated based on information on a first position corresponding to the work implement on the surface of the work object opposite the work implement, and information on the position of a portion extending from the first position along the direction in which the rotating body rotates.

[0015] According to a tenth aspect, there is provided a remote control system for a work machine, including the image display system for a work machine according to the ninth aspect, and an operation device for operating the work implement provided on the work machine.

[0016] According to an eleventh aspect, there is provided a work machine equipped with the work machine image display system according to any one of the first to ninth aspects.

[0017] According to a twelfth aspect, there is provided an image display method for a work machine that is applicable to a work machine that includes a work implement, a work machine having the work implement, a rotating body to which the work implement is attached, and an imaging device attached to the rotating body, the method including the steps of: using information on the position of the work implement obtained using the attitude of the work implement and information on the position of the work object obtained from information on the distance from the work machine to the work object, generating a first image on a part of the work implement based on information on a corresponding first position on the surface of the work object facing the work implement and information on the position of a portion extending from the first position along the direction in which the rotating body rotates; and displaying the generated first image on a display device, wherein the first image is generated based on information on the first position on the surface of the work object facing the work implement, which corresponds to the work implement, and information on the position of a portion extending from the first position along the direction in which the rotating body rotates. [Effects of the Invention]

[0018] The present invention can suppress a decrease in work efficiency when working using a work machine equipped with a work implement having a work tool. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram showing a construction machine image display system and a construction machine remote control system according to an embodiment. [Figure 2] FIG. 2 is a diagram showing a control system of a hydraulic excavator, which is a work machine according to the embodiment. [Figure 3] FIG. 3 is a diagram for explaining the coordinate systems of the image display system and the remote control system according to the embodiment. [Figure 4] FIG. 4 is a rear view of the hydraulic excavator. [Figure 5]FIG. 5 is a diagram illustrating the coordinate systems of the imaging device and the distance detection device. [Figure 6] FIG. 6 is a flowchart of an example of control executed by the image display system and the remote control system according to the embodiment. [Figure 7] FIG. 7 is a diagram showing an imaging device, a distance detection device, and a work object. [Figure 8] FIG. 8 is a diagram illustrating an occupied area. [Figure 9] FIG. 9 is a diagram showing information on the shape of the work object from which the occupied area has been removed. [Figure 10] FIG. 10 is a diagram showing an example of the first image. [Figure 11] FIG. 11 is a diagram for explaining the process in which the processing device generates the first image. [Figure 12] FIG. 12 is a diagram for explaining the process in which the processing device generates the first image. [Figure 13] FIG. 13 is a diagram showing the positional relationship between the origin of the imaging device and the first image, the first linear image, and the second linear image. [Figure 14] FIG. 14 is a diagram showing the reference image and the first image shown in FIG. [Figure 15] FIG. 15 shows a working image. [Figure 16] FIG. 16 is a diagram showing a reference image according to a modified example. [Figure 17] FIG. 17 is a diagram showing an example of the first image and the third image. [Figure 18] FIG. 18 is a diagram showing a control system of a hydraulic excavator according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0020] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail with reference to the drawings.

[0021] <Outline of the work machine image display system and work machine remote control system> FIG. 1 is a diagram showing an image display system 100 for a work machine and a remote control system 101 for a work machine according to an embodiment. When an operator remotely controls a hydraulic excavator 1, which is a work machine, the image display system 100 for a work machine uses an imaging device 19 to capture an image of a work object WA and a bucket 8, which is a work implement, of the hydraulic excavator 1, and displays the obtained image on a display device 52. At this time, the image display system 100 displays, on the display device 52, a work image 69 including an image 60 indicating the position of the bucket 8 on the work object WA and an image 68 of the work object WA captured by the imaging device 19. The image 68 also includes an image of the bucket 8. In the embodiment, the work image 69 further includes a reference image 65. The reference image 65 is an image displayed along the surface of the work object WA. The reference image 65 serves as an indicator indicating the position of the work object WA. The work object of the hydraulic excavator 1 is the terrain surface on which work is to be performed by a work implement 2 provided on the hydraulic excavator 1, i.e., the work object WA.

[0022] The image display system 100 includes an imaging device 19, an attitude detection device 32, a distance detection device 20, and a processing device 51. A work machine remote control system 101 (hereinafter referred to as the remote control system 101 as appropriate) includes the imaging device 19, the attitude detection device 32, the distance detection device 20, a work machine control device 27, a display device 52, the processing device 51, and an operation device 53. In the embodiment, the imaging device 19, the attitude detection device 32, and the distance detection device 20 of the image display system 100 are provided in the hydraulic excavator 1, and the processing device 51 is provided in a facility 50. The facility 50 is used to remotely control and manage the hydraulic excavator 1. In the embodiment, the imaging device 19, the attitude detection device 32, the distance detection device 20, and the work machine control device 27 of the remote control system 101 are provided in the hydraulic excavator 1, and the display device 52, the processing device 51, and the operation device 53 are provided in the facility 50.

[0023] The processing device 51 of the image display system 100 includes a processing unit 51P, a memory unit 51M, and an input / output unit 51IO. The processing unit 51P is exemplified by a processor such as a CPU (Central Processing Unit). The memory unit 51M is exemplified by a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk drive, a storage device, or a combination thereof. The input / output unit 51IO is an interface circuit for connecting the processing device 51 to external devices. In the embodiment, the input / output unit 51IO is connected to a display device 52, an operation device 53, and a communication device 54 as external devices. The external devices connected to the input / output unit 51IO are not limited to these.

[0024] The processing device 51 uses information on the position of the bucket 8 obtained using the attitude of the work implement 2 and information on the position of the work object WA obtained from the distance information determined by the distance detection device 20 to generate a first image based on information on a first position corresponding to the bucket 8 on the work object WA facing the bucket 8 and information on the position of a portion extending from the first position along the direction in which the upper rotating body 3, which is a rotating body, rotates, using the imaging device 19 as a reference. The first position may be a position corresponding to a portion of the bucket 8 on the work object WA facing the bucket 8. The processing device 51 combines the first image with an image 68 of the work object WA captured by the imaging device 19 and displays the combined image on the display device 52. The work object WA is a surface on which the work implement 2 of the hydraulic excavator 1 performs work such as excavation or earth leveling. The first image is included in an image 60 for indicating the position of the bucket 8 on the work object WA. Hereinafter, the image 68 will be referred to as a second image 68 as appropriate.

[0025] The display device 52 is exemplified by, but not limited to, a liquid crystal display or a projector. The communication device 54 is provided with an antenna 54A. The communication device 54 communicates with the communication device 25 provided in the hydraulic excavator 1 to obtain information about the hydraulic excavator 1 and transmit information to the hydraulic excavator 1.

[0026] The operating device 53 has a left operating lever 53L installed on the left side of the operator and a right operating lever 53R installed on the right side of the operator. The left operating lever 53L and the right operating lever 53R are configured so that forward, backward, left and right operations correspond to the movements of two axes. For example, forward and backward operation of the right operating lever 53R corresponds to the movement of the boom 6 of the work implement 2 provided in the hydraulic excavator 1. Left and right operation of the right operating lever 53R corresponds to the movement of the bucket 8 of the work implement 2. Forward and backward operation of the left operating lever 53L corresponds to the movement of the arm 7 of the work implement 2. Left and right operation of the left operating lever 53L corresponds to the rotation of the upper rotating body 3 of the hydraulic excavator 1.

[0027] The amounts of operation of the left operating lever 53L and the right operating lever 53R are detected by, for example, a potentiometer and a Hall IC, and the processing device 51 generates a control signal for controlling the electromagnetic control valve based on these detected values. This signal is sent to the work implement control device 27 via the communication device 54 of the facility 50 and the communication device 25 of the hydraulic excavator 1. The work implement control device 27 controls the electromagnetic control valve based on the control signal, thereby controlling the work implement 2. The electromagnetic control valve will be described later.

[0028] The processing device 51 acquires input to at least one of the left operating lever 53L and the right operating lever 53R, and generates a command for operating at least one of the work implement 2 and the upper rotating body 3. The processing device 51 transmits the generated command to the communication device 25 of the hydraulic excavator 1 via the communication device 54. The work implement control device 27 provided in the hydraulic excavator 1 acquires the command from the processing device 51 via the communication device 25, and operates at least one of the work implement 2 and the upper rotating body 3 in accordance with the command.

[0029] The hydraulic excavator 1 is equipped with a communication device 25, a work implement control device 27, an attitude detection device 32, an imaging device 19, a distance detection device 20, antennas 21 and 22, and a position calculation device 23. The work implement control device 27 controls the work implement 2. The communication device 25 is connected to an antenna 24 and communicates with a communication device 54 provided in the facility 50. The work implement control device 27 controls the work implement 2 and the upper rotating body 3. The attitude detection device 32 detects the attitude of at least one of the work implement 2 and the hydraulic excavator 1. The imaging device 19 is attached to the hydraulic excavator 1 and captures an image of the work object WA. The distance detection device 20 obtains information about the distance from a predetermined position of the hydraulic excavator 1 to the work object WA. The antennas 21 and 22 receive radio waves from a positioning satellite 200. The position calculation device 23 obtains the global positions of the antennas 21 and 22, i.e., their positions in a global coordinate system, using the radio waves received by the antennas 21 and 22.

[0030] <Overall configuration of hydraulic excavator 1> The hydraulic excavator 1 has a vehicle body 1B as the main body and a work implement 2. The vehicle body 1B has an upper rotating body 3 and a traveling device 5 as a traveling body. The upper rotating body 3 has a driver's cab 4. The traveling device 5 mounts the upper rotating body 3. The traveling device 5 has tracks. The traveling device 5 causes the hydraulic excavator 1 to travel by rotating the tracks. The work implement 2 is attached to the side of the driver's cab 4 of the upper rotating body 3.

[0031] The side of the upper rotating body 3 on which the work implement 2 and the operator's cab 4 are located is the front, and the side on which the counterweight 9 is located is the rear. The left side as you face the front is the left of the upper rotating body 3, and the right side as you face the front is the right of the upper rotating body 3. With respect to the hydraulic excavator 1 or the vehicle body 1B, the traveling gear 5 side is the bottom relative to the upper rotating body 3, and the upper rotating body 3 side is the top relative to the traveling gear 5. When the hydraulic excavator 1 is installed on a horizontal surface, the bottom is the vertical direction, i.e., the side in the direction of gravity, and the top is the opposite side to the vertical direction.

[0032] The work implement 2 has a boom 6, an arm 7, and working implements that include a bucket 8, a boom cylinder 10, an arm cylinder 11, and a bucket cylinder 12. The bucket 8 has multiple blades 8B. A cutting edge 8T is the tip of the blade 8B. The bucket 8 is not limited to one that has multiple blades 8B. The bucket 8 may also be a tilt bucket. Alternatively, the work implement 2 may be equipped with a slope bucket or a rock drilling attachment equipped with a rock drilling tip as a working implement instead of the bucket 8.

[0033] The boom cylinder 10, arm cylinder 11, and bucket cylinder 12 are hydraulic cylinders that are driven by the pressure of hydraulic oil discharged from a hydraulic pump. The boom cylinder 10 drives the boom 6. The arm cylinder 11 drives the arm 7. The bucket cylinder 12 drives the bucket 8.

[0034] Antennas 21, 22 and an antenna 24 are attached to the top of the upper rotating body 3. The antennas 21, 22 are used to detect the position of the hydraulic excavator 1. The antennas 21, 22 are electrically connected to a position calculation device 23.

[0035] The antennas 21 and 22 are antennas for GNSS (Kinematic-Global Navigation Satellite Systems). The antennas 21 and 22 are arranged a certain distance apart in a direction parallel to the width direction of the upper rotating body 3. The antennas 21 and 22 receive GNSS radio waves from a positioning satellite 200 and output signals corresponding to the received GNSS radio waves. The antennas 21 and 22 may also be antennas for a GPS (Global Positioning System). In the following description, the antennas 21 and 22 will be referred to as GNSS antennas 21 and 22 as appropriate. The position calculation device 23 detects the position of the hydraulic excavator 1, for example, by using the GNSS.

[0036] 1 , and the distance detection device 20 determines the distance from itself (a predetermined position of the hydraulic excavator 1) to the work object WA, it is preferable to acquire information from as wide an area of ​​the work object WA as possible. For this reason, in the embodiment, the antenna 24, the image capture device 19, and the distance detection device 20 are installed above the operator's cab 4 of the upper rotating body 3. The location where the image capture device 19 and the distance detection device 20 are installed is not limited to above the operator's cab 4. For example, the image capture device 19 and the distance detection device 20 may be installed inside and above the operator's cab 4.

[0037] The imaging device 19 has an incident surface 19L facing forward of the upper rotating body 3. The distance detection device 20 has an incident surface 20L facing forward of the upper rotating body 3. In the embodiment, the imaging device 19 is a monocular camera equipped with an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. In the embodiment, the distance detection device 20 is a 3D laser range finder or a distance sensor. The imaging device 19 and the distance detection device 20 are not limited to these. For example, instead of the imaging device 19 and the distance detection device 20, a device having both the function of acquiring an image of the work object WA and the function of calculating the distance to the work object WA may be used. An example of such a device is a stereo camera.

[0038] <Hydraulic excavator 1 control system> 2 is a diagram showing a control system 1S of a hydraulic excavator 1, which is a work machine according to an embodiment. The control system 1S includes a communication device 25, a sensor controller 26, a work machine control device 27, an imaging device 19, a distance detection device 20, a position calculation device 23, an attitude detection device 32, an IMU (Inertial Measurement Unit) 33, and a hydraulic system 36. The communication device 25, the sensor controller 26, and the work machine control device 27 are connected by a signal line 35. With this structure, the communication device 25, the sensor controller 26, and the work machine control device 27 can exchange information with each other via the signal line 35. An example of the signal line that transmits information within the control system 1S is an in-vehicle signal line such as a CAN (Controller Area Network).

[0039] The sensor controller 26 has a processor such as a CPU and storage devices such as a RAM and a ROM. The sensor controller 26 receives the detection values ​​of the position calculation device 23, information on the image captured by the imaging device 19, the detection values ​​of the distance detection device 20, the detection values ​​of the attitude detection device 32, and the detection values ​​of the IMU 33. The sensor controller 26 transmits the input detection values ​​and image information to a processing device 51 in the facility 50 shown in FIG. 1 via a signal line 35 and the communication device 25.

[0040] The work implement control device 27 has a processor such as a CPU and storage devices such as a RAM and a ROM. The work implement control device 27 acquires, via the communication device 25, commands for operating at least one of the work implement 2 and the upper rotating body 3, which commands have been generated by a processing device 51 of the facility 50. The work implement control device 27 controls the electromagnetic control valve 28 of the hydraulic system 36 based on the acquired commands.

[0041] The hydraulic system 36 includes an electromagnetic control valve 28, a hydraulic pump 29, and hydraulic actuators such as the boom cylinder 10, the arm cylinder 11, the bucket cylinder 12, and the swing motor 30. The hydraulic pump 29 is driven by an engine 31 and discharges hydraulic oil for operating the hydraulic actuators. The work implement control device 27 controls the flow rate of hydraulic oil supplied to the boom cylinder 10, the arm cylinder 11, the bucket cylinder 12, and the swing motor 30 by controlling the electromagnetic control valve 28. In this way, the work implement control device 27 controls the operation of the boom cylinder 10, the arm cylinder 11, the bucket cylinder 12, and the swing motor 30.

[0042] The sensor controller 26 acquires the detection values ​​of the first stroke sensor 16, the second stroke sensor 17, and the third stroke sensor 18. The first stroke sensor 16 is provided on the boom cylinder 10, the second stroke sensor 17 on the arm cylinder 11, and the third stroke sensor 18 on the bucket cylinder 12.

[0043] The first stroke sensor 16 detects the boom cylinder length, which is the length of the boom cylinder 10, and outputs the result to the sensor controller 26. The second stroke sensor 17 detects the arm cylinder length, which is the length of the arm cylinder 11, and outputs the result to the sensor controller 26. The third stroke sensor 18 detects the bucket cylinder length, which is the length of the bucket cylinder 12, and outputs the result to the sensor controller 26.

[0044] Once the boom cylinder length, arm cylinder length, and bucket cylinder length are determined, the attitude of the work implement 2 is determined. Therefore, the first stroke sensor 16, second stroke sensor 17, and third stroke sensor 18 that detect these lengths correspond to the attitude detection device 32 that detects the attitude of the work implement 2. The attitude detection device 32 is not limited to the first stroke sensor 16, second stroke sensor 17, and third stroke sensor 18, and may be an angle detector.

[0045] The sensor controller 26 calculates the inclination angle of the boom 6 with respect to a direction perpendicular to the horizontal plane in the vehicle body coordinate system, which is the coordinate system of the hydraulic excavator 1, from the boom cylinder length detected by the first stroke sensor 16. The sensor controller 26 calculates the inclination angle of the arm 7 with respect to the boom 6 from the arm cylinder length detected by the second stroke sensor 17. The sensor controller 26 calculates the inclination angle of the bucket 8 with respect to the arm 7 from the bucket cylinder length detected by the third stroke sensor 18. The inclination angles of the boom 6, arm 7, and bucket 8 are information that indicates the attitude of the work implement 2. In other words, the sensor controller 26 obtains information that indicates the attitude of the work implement 2. The sensor controller 26 transmits the calculated inclination angle to a processing device 51 of the facility 50 shown in FIG. 1 via a signal line 35 and the communication device 25.

[0046] The position calculation device 23 has a processor such as a CPU and storage devices such as a RAM and a ROM. The position calculation device 23 determines the position of the hydraulic excavator 1. In detail, the position calculation device 23 detects and outputs the positions of the antennas 21 and 22 and the orientation of the upper rotating body 3 in the global coordinate system using signals acquired from the antennas 21 and 22. The orientation of the upper rotating body 3 represents the direction of the upper rotating body 3 in the global coordinate system. The orientation of the upper rotating body 3 can be represented, for example, by the direction of the fore-and-aft direction of the upper rotating body 3 around the vertical axis of the global coordinate system. The azimuth angle is the rotation angle of a reference axis in the fore-and-aft direction of the upper rotating body 3 around the vertical axis of the global coordinate system. The orientation of the upper rotating body 3 is represented by the azimuth angle.

[0047] The IMU 33 detects the movement and attitude of the hydraulic excavator 1. The movement of the hydraulic excavator 1 includes at least one of the movement of the upper rotating body 3 and the movement of the traveling unit 5. The attitude of the hydraulic excavator 1 is represented by the roll angle, pitch angle, and yaw angle of the hydraulic excavator 1. In the embodiment, the IMU 33 detects and outputs the angle or angular velocity, and acceleration of the hydraulic excavator 1. The angles of the hydraulic excavator 1 detected by the IMU 33 are the roll angle, pitch angle, and yaw angle of the hydraulic excavator 1.

[0048] <About the coordinate system> Fig. 3 is a diagram for explaining the coordinate systems of the image display system 100 and the remote control system 101 according to the embodiment. Fig. 4 is a rear view of the hydraulic excavator 1. Fig. 5 is a diagram for explaining the coordinate systems of the imaging device 19 and the distance detection device 20.

[0049] The image display system 100 and the remote control system 101 include a global coordinate system, a vehicle body coordinate system, a coordinate system of the image capture device 19, and a coordinate system of the distance detection device 20. The global coordinate system (Xg, Yg, Zg) is a three-dimensional coordinate system based on an origin PG fixed to the earth. In the embodiment, the global coordinate system is, for example, a coordinate system in GNSS or GPS.

[0050] The vehicle body coordinate system is a three-dimensional coordinate system expressed as (Xm, Ym, Zm) with the hydraulic excavator 1 as the reference. In the embodiment, the origin position PL of the vehicle body coordinate system is the intersection of the Zm axis, which is the rotation center axis of the upper rotating body 3, and a plane perpendicular to the Zm axis within the swing circle of the upper rotating body 3, but is not limited to this. The Xm axis extends in the front-rear direction of the upper rotating body 3 and is perpendicular to the Zm axis. The Xm axis is a reference axis in the front-rear direction of the upper rotating body 3. The Ym axis extends in the width direction of the upper rotating body 3 and is perpendicular to the Zm axis and Xm axis. The plane perpendicular to the Zm axis within the swing circle can be a plane passing through the center of the swing circle in the Zm axis direction.

[0051] In the embodiment, the coordinate system of the imaging device 19 (hereinafter referred to as the imaging device coordinate system as appropriate) is a three-dimensional coordinate system represented by (Xc, Yc, Zc) with the center of the imaging surface 19P of the imaging element 19RC as the origin PC, as shown in Fig. 5. The Xc axis of the imaging device coordinate system (Xc, Yc, Zc) is an axis that passes through the optical center of the imaging device 19 and extends in a direction perpendicular to the imaging surface 19P. The Yc axis is an axis perpendicular to the Xc axis. The Zc axis is an axis perpendicular to both the Xc axis and the Yc axis.

[0052] In this embodiment, the coordinate system of the distance detection device 20 is a three-dimensional coordinate system represented by (Xd, Yd, Zd) with the center of the detection surface 20P of the distance detection element 20RC as the origin PD, as shown in FIG.

[0053] <Posture of hydraulic excavator 1> 4, the inclination angle θ4 of the upper rotating body 3 in the left-right direction, i.e., the width direction, is the roll angle of the hydraulic excavator 1, the inclination angle θ5 of the upper rotating body 3 in the front-rear direction is the pitch angle of the hydraulic excavator 1, and the angle of the upper rotating body 3 around the vertical axis is the yaw angle of the hydraulic excavator 1. The roll angle, pitch angle, and yaw angle can be found by integrating the angular velocity detected by the IMU 33 over time.

[0054] The acceleration and angular velocity detected by the IMU 33 are output as operation information to the sensor controller 26. The sensor controller 26 performs processing such as filtering and integration on the operation information acquired from the IMU 33 to determine the inclination angle θ4, which is the roll angle, the inclination angle θ5, which is the pitch angle, and the yaw angle. The sensor controller 26 transmits the determined inclination angle θ4, inclination angle θ5, and yaw angle as information related to the attitude of the hydraulic excavator 1 to a processing device 51 in a facility 50 shown in FIG. 1 via a signal line 35 and a communication device 25 shown in FIG. 2.

[0055] As described above, the sensor controller 26 obtains information indicating the attitude of the work implement 2. Specifically, the information indicating the attitude of the work implement 2 is the tilt angle θ1 of the boom 6 with respect to the direction (Zm axis direction) perpendicular to the horizontal plane in the vehicle body coordinate system, the tilt angle θ2 of the arm 7 with respect to the boom 6, and the tilt angle θ3 of the bucket 8 with respect to the arm 7. The processing device 51 of the facility 50 shown in FIG. 1 calculates the position P4 of the cutting edge 8T of the bucket 8 (hereinafter referred to as the cutting edge position as appropriate) from the information indicating the attitude of the work implement 2 obtained from the sensor controller 26 of the hydraulic excavator 1, i.e., the tilt angles θ1, θ2, and θ3.

[0056] The memory unit 51M of the processing device 51 stores data on the work machine 2 (hereinafter referred to as work machine data as appropriate). The work machine data includes the length L1 of the boom 6, the length L2 of the arm 7, and the length L3 of the bucket 8. As shown in FIG. 3, the length L1 of the boom 6 corresponds to the length from the boom pin 13 to the arm pin 14. The length L2 of the arm 7 corresponds to the length from the arm pin 14 to the bucket pin 15. The length L3 of the bucket 8 corresponds to the length from the bucket pin 15 to the cutting edge 8T of the bucket 8. The cutting edge 8T is the tip of the blade 8B shown in FIG. 1. The work machine data includes information on the position of the boom pin 13 relative to the origin position PL of the vehicle body coordinate system. The processing device 51 can determine the cutting edge position P4 relative to the origin position PL using the lengths L1, L2, L3, the inclination angles θ1, θ2, θ3, and the origin position PL. In the embodiment, the processing device 51 of the facility 50 determines the cutting edge position P4, but the sensor controller 26 of the hydraulic excavator 1 may determine the cutting edge position P4 and transmit it to the processing device 51 of the facility 50. The cutting edge position P4 is the position of the cutting edge 8T of the bucket 8, which is part of the work implement 2, and is therefore the position of the work implement 2. In the former case, the attitude detection device 32, the sensor controller 26, and the processing device 51 correspond to a position detection unit that detects the position of the work implement. In the latter case, the attitude detection device 32 and the sensor controller 26 correspond to a position detection unit that detects the position of the work implement.

[0057] The position detection unit is not limited to the above-described attitude detection device 32, sensor controller 26, and processing device 51, or the attitude detection device 32 and sensor controller 26. For example, the position of the work machine may be detected by a distance measurement device such as a stereo camera or a laser scanner. In this case, the distance measurement device corresponds to the position detection unit that detects the position of the work machine.

[0058] <Example of control performed by the image display system 100 and the remote control system 101> Fig. 6 is a flowchart of an example of control executed by the image display system 100 and the remote control system 101 according to the embodiment. Fig. 7 is a diagram showing the image capturing device 19, the distance detecting device 20, and the work object WA.

[0059] In step S101, the sensor controller 26 shown in FIG. 2 acquires information about the hydraulic excavator 1. The information about the hydraulic excavator 1 is information obtained from the imaging device 19, distance detection device 20, position calculation device 23, attitude detection device 32, and IMU 33. As shown in FIG. 7, the imaging device 19 captures an image of the work object WA within an imaging range TA to obtain an image of the work object WA. The distance detection device 20 detects the distance Ld from the distance detection device 20 to the work object WA and other objects present within the detection range MA. The position calculation device 23 obtains reference position information Pga1 and Pga2 corresponding to positions P1 and P2 of the GNSS antennas 21 and 22 in the global coordinate system. The attitude detection device 32 detects the boom cylinder length, arm cylinder length, and bucket cylinder length. The IMU 33 detects the attitude of the hydraulic excavator 1; more specifically, the roll angle θ4, pitch angle θ5, and yaw angle of the upper rotating body 3.

[0060] In step S102, the image display system 100 and the processing device 51 of the remote control system 101 acquire information about the hydraulic excavator 1 from the sensor controller 26 of the hydraulic excavator 1 via the communication device 25 of the hydraulic excavator 1 and the communication device 54 connected to the processing device 51.

[0061] The information of the hydraulic excavator 1 that the processing device 51 acquires from the sensor controller 26 includes an image of the work object WA captured by the imaging device 19, information on the distance from the distance detection device 20 to the work object WA detected by the distance detection device 20, information on the attitude of the work implement 2 provided on the hydraulic excavator 1 detected by the attitude detection device 32, reference position information Pga1, Pga2, and information on the attitude of the hydraulic excavator 1.

[0062] Information about the distance from the distance detection device 20 to the work object WA includes information about the distance Ld to the work object WA or object OB present within the detection range MA and information about the orientation of a position Pd corresponding to the distance Ld. In the example shown in FIG. 7, the distance Ld is shown as the distance to the work object WA. Information about the orientation of the position Pd is the orientation of the position Pd relative to the distance detection device 20, and is the angle relative to the axes Xd, Yd, and Zd of the coordinate system of the distance detection device 20. Information about the attitude of the work implement 2 acquired by the processing device 51 is the inclination angles θ1, θ2, and θ3 of the work implement 2 calculated by the sensor controller 26 using the boom cylinder length, arm cylinder length, and bucket cylinder length. Information about the attitude of the hydraulic excavator 1 is the roll angle θ4, pitch angle θ5, and yaw angle of the hydraulic excavator 1, more specifically, of the upper rotating body 3.

[0063] Processing device 51 uses tilt angles θ1, θ2, and θ3 of work implement 2 acquired from sensor controller 26, and length L1 of boom 6, length L2 of arm 7, and length L3 of bucket 8 stored in memory unit 51M to determine cutting edge position P4 of bucket 8. Cutting edge position P4 of bucket 8 is a set of coordinates in the vehicle body coordinate system (Xm, Ym, Zm) of hydraulic excavator 1.

[0064] In step S103, the processing device 51 converts the distance Ld to the work object WA into position information using information about the distance to the work object WA. The position information is the coordinates of the position Pd in ​​the coordinate system (Xd, Yd, Zd) of the distance detection device 20. In step S103, all distances Ld detected by the distance detection device 20 within the detection range MA are converted into position information. The processing device 51 converts the distance Ld into position information using the distance Ld and information about the direction of the position Pd corresponding to the distance Ld. In step S103, the distance to an object OB present within the detection range MA is also converted into position information, similar to the distance Ld to the work object WA. By the processing of step S103, information about the position of the work object WA within the detection range MA can be obtained. Information about the shape of the work object WA can be obtained from the information about the position of the work object WA.

[0065] The position information and shape information of the work object WA are a set of coordinates of the position Pd in ​​the coordinate system (Xd, Yd, Zd) of the distance detection device 20. The processing device 51 converts the shape information of the work object WA into values ​​in the imaging device coordinate system (Xc, Yc, Zc), and then converts them into values ​​in the vehicle body coordinate system (Xm, Ym, Zm). Next, in step S104, the processing device 51 determines the occupied area SA.

[0066] FIG. 8 is a diagram illustrating the occupation area SA. The occupation area SA is the area occupied by the work implement 2 within the information on the shape of the work object WA. In the example shown in FIG. 8, the bucket 8 of the work implement 2 is within the detection range MA of the distance detection device 20, and is between the distance detection device 20 and the work object WA. For this reason, the distance detection device 20 detects the distance to the bucket 8, not the distance to the work object WA, for the portion of the occupation area SA. In step S105, the processing device 51 removes the portion of the occupation area SA from the information on the shape of the work object WA obtained in step S103.

[0067] When removing the portion of the occupied area SA, the processing device 51 stores, for example, in the memory unit 51M, information on at least one of the position and attitude of the bucket 8 detected by the distance detection device 20 according to at least one of the position and attitude of the bucket 8. In this embodiment, such information is included in the attitude of the work implement 2 of the hydraulic excavator 1. The attitude of the work implement 2 can be determined using the tilt angles θ1, θ2, and θ3 of the work implement 2, the length L1 of the boom 6, the length L2 of the arm 7, and the length L3 of the bucket 8, and, if necessary, the attitude of the hydraulic excavator 1. The processing device 51 then compares the information detected by the distance detection device 20 with the information stored in the memory unit 51M, and if the two match, it can be determined that the bucket 8 has been detected. Since the portion where the two match corresponds to the occupied area SA, the processing device 51 removes the portion of the occupied area SA from the information on the shape of the work object WA.

[0068] By using the posture of the work machine 2 to remove the occupied area SA, the processing device 51 does not use information about the bucket 8 in the occupied area SA when generating the reference image 65, which serves as an indicator representing the position of the work object WA shown in Figure 1, and therefore can accurately generate the reference image 65.

[0069] The process of removing a portion of the occupied area SA using the attitude of the work implement 2 may be performed by the following method. Information regarding at least one of the position and attitude of the bucket 8 in the vehicle body coordinate system, which is included in the attitude of the work implement 2, is obtained from the tilt angles θ1, θ2, θ3 of the work implement 2, the length L1 of the boom 6, the length L2 of the arm 7, and the length L3 of the bucket 8. In step S103, information about the shape of the work object WA in the vehicle body coordinate system is obtained. In step S105, the processing device 51 removes from the shape of the work object WA an area obtained by projecting the position of the bucket 8 onto the information about the shape of the work object WA, as the occupied area SA.

[0070] FIG. 9 is a diagram showing information about the shape of the work object WA with the occupied area removed. Information IMWA about the shape of the work object WA is a set of coordinates Pmd (Xmd, Ymd, Zmd) in the vehicle body coordinate system (Xm, Ym, Zm). As a result of the processing in step S105, no coordinate information exists for the occupied area SA. Next, proceeding to step S106, the processing device 51 generates a first image. The first image is an image that includes a portion of the work object WA that corresponds to the bucket 8 or a portion of the bucket 8, and extends along the direction in which the upper rotating body 3 of the hydraulic excavator 1 rotates.

[0071] <First image> Fig. 10 is a diagram showing an example of the first image IMC. Figs. 11 and 12 are diagrams for explaining the process by which the processing device 51 generates the first image IMC. In the embodiment, the first image IMC shown in Fig. 10 shows the position of the cutting edge 8T of the bucket 8 on the work object WA, and also shows the position of the cutting edge 8T on the surface of the work object WA when the upper rotating body 3 rotates while the work implement 2 maintains its current posture. Next, the process by which the processing device 51 generates the first image IMC will be explained.

[0072] 10 and 12, when viewed from the Zm axis of the vehicle body coordinate system, which is the central axis of rotation of the upper rotating body 3, the first image IMC is an image of an arc having a radius equal to the distance Rbt between the Zm axis and the bucket 8 and a center at a position corresponding to the Zm axis. In the embodiment, the position of the bucket 8 used in calculating the distance Rbt is the position of the cutting edge 8T of the bucket 8 that is located at the center of the width direction Wb of the bucket 8, as shown in FIG. 12. The position of the bucket 8 used in calculating the distance Rbt is not limited to the cutting edge 8T at the center of the width direction Wb of the bucket 8. The distance Rbt is the distance between the Zm axis of the vehicle body coordinate system and the position Pbt(Xmbt, Ymbt, Zmbt) of the cutting edge 8T at the center of the width direction Wb of the bucket 8, along a direction parallel to the Xm-Ym plane of the vehicle body coordinate system. The position Pbt(Xmbt, Ymbt, Zmbt) is a position in the vehicle body coordinate system.

[0073] Line LC in Figure 12 indicates the center of the width direction Wb of the bucket 8. The width direction Wb of the bucket 8 is parallel to the direction in which bucket pin 15 that connects the bucket 8 to the arm 7 extends, and is parallel to the Ym-axis of the vehicle body coordinate system. Distance Rbt is the distance between the Zm-axis and the cutting edge 8T of the bucket 8 along a direction parallel to the Xm-axis in the vehicle body coordinate system.

[0074] The position indicated by the first image IMC is the position of the surface WAS of the work object WA. The processing device 51 determines the position Pt(Xmt, Ymt, Zmt) of the portion (hereinafter referred to as the "intersection portion") obtained by projecting an arc with a radius of distance Rbt centered on the Zm axis of the vehicle body coordinate system onto the surface WAS of the work object WA in a direction parallel to the Zm axis of the vehicle body coordinate system. The image of the intersection portion becomes the first image IMC.

[0075] The position of the intersection Pt(Xmt, Ymt, Zmt) is, for example, the position where a curved surface having a radius of a distance Rbt centered on the Zm axis of the vehicle body coordinate system intersects with the surface WAS of the work object WA. The position of part of the intersection is the point where a straight line VL passing through the cutting edge 8T of the bucket 8 located at the center of the width direction Wb of the bucket 8 and extending in a direction parallel to the Zm axis of the vehicle body coordinate system intersects with the surface WAS of the work object WA. The processing device 51 uses information about the position of the intersection Pt(Xmt, Ymt, Zmt) to generate a first image IMC. The first image IMC corresponds to the information about the position of the intersection Pt(Xmt, Ymt, Zmt).

[0076] A first position Ptc among the intersection positions Pt (Xmt, Ymt, Zmt) corresponds to the position of the bucket 8 on the surface WAS of the work object WA facing the bucket 8, and in this embodiment corresponds to the position of the cutting edge 8T of the bucket 8 located at the center in the width direction Wb of the bucket 8. The first position Ptc is not limited to the position of the cutting edge 8T at the center in the width direction Wb, and may be, for example, a position at one end of the width direction Wp.

[0077] The position of the portion extending from the first position Ptc along the rotation direction Rtd of the upper rotating body 3 is the position of the portion of the intersection position Pt(Xmt, Ymt, Zmt) extending from the first position Ptc along the rotation direction Rtd. The first image IMC corresponds to information on the first position Ptc and information on the position of the portion extending from the first position Ptc along the rotation direction Rtd. In other words, the first image IMC is generated based on information on the first position Ptc and information on the position of the portion extending from the first position Ptc along the rotation direction Rtd.

[0078] In this embodiment, the processing device 51 also displays a first line image 62, which is an image of the line LV1, and a second line image 63, which is an image of the line LV2, on the display device 52 shown in FIG. 1. The line LV1 is a line extending from an outer position Pb1 of the blade 8B at one end 8Wt1 of the bucket 8 in the width direction Wb, along a direction parallel to the Zm axis of the vehicle body coordinate system, to the surface WAS of the work object WA. The line LV2 is a line extending from an outer position Pb2 of the blade 8B at the other end 8Wt2 of the bucket 8 in the width direction Wb, along a direction parallel to the Zm axis of the vehicle body coordinate system, to the surface WAS of the work object WA. The position where the line LV1 intersects with the surface WAS of the work object WA is designated as Pt1, and the position where the line LV2 intersects with the surface WAS of the work object WA is designated as Pt2. Since the position of each part of the bucket 8 in the vehicle body coordinate system and the position of the surface WAS of the work object WA in the vehicle body coordinate system are known, the processing device 51 can determine positions Pb1, Pb2, Pt1, and Pt2 from these positions. Once positions Pb1, Pb2, Pt1, and Pt2 are obtained, lines LV1 and LV2 are also obtained, allowing the processing device 51 to generate a first line image 62 and a second line image 63.

[0079] Once the first image IMC, the first linear image 62, and the second linear image 63 are obtained, the processing device 51 converts them into an image from the viewpoint of the imaging device 19. The image from the viewpoint of the imaging device 19 is an image based on the imaging device 19. Next, the process for converting the first image IMC, the first linear image 62, and the second linear image 63 into an image from the viewpoint of the imaging device 19 will be described.

[0080] 13 is a diagram showing the positional relationship between the origin PC of the imaging device 19 and the first image IMC, the first straight line image 62, and the second straight line image 63. The image at the viewpoint of the imaging device 19 is an image obtained when the first image IMC, the first straight line image 62, and the second straight line image 63 are viewed from the origin PC of the imaging device in the vehicle body coordinate system (Xm, Ym, Zm).

[0081] The first image IMC, the first linear image 62, and the second linear image 63 are images in a three-dimensional space, but the image from the viewpoint of the imaging device 19 is a two-dimensional image. Therefore, the processing device 51 performs perspective projection transformation to project the first image IMC, the first linear image 62, and the second linear image 63, which are defined in the three-dimensional space, i.e., the vehicle body coordinate system (Xm, Ym, Zm), onto a two-dimensional surface. The first image IMC, the first linear image 62, and the second linear image 63 converted into images from the viewpoint of the imaging device 19 will be referred to as a guide image 60 below as appropriate.

[0082] 14 is a diagram showing the reference image 65 and first image IMC shown in FIG. 1. After the guide image 60 is generated, the process proceeds to step S107, where the processing device 51 generates the reference image 65. The reference image 65 is a line image along the surface WAS of the work object WA. The processing device 51 generates the reference image 65 using information about the position of the work object WA, more specifically, information about the position of the work object WA on the surface WAS.

[0083] The reference image 65 includes a plurality of first line images RL extending radially from a position corresponding to the Zm axis, which is the central axis of rotation of the upper rotating body 3, and a plurality of second line images CL extending along the rotation direction Rtd of the upper rotating body 3 with the Zm axis as the center. In the embodiment, the plurality of first line images RL are line images that extend radially from the Zm axis and are arranged along the circumferential direction of a circle centered on the Zm axis, when viewed from the Zm axis direction of the vehicle body coordinate system (Xm, Ym, Zm). The plurality of second line images CL are images of arcs or circles that extend along the circumferential direction of a circle centered on the Zm axis, when viewed from the Zm axis direction of the vehicle body coordinate system (Xm, Ym, Zm), and are arranged along the radial direction of the circle centered on the Zm axis.

[0084] The first line image RL and the second line image CL are defined in the vehicle body coordinate system (Xm, Ym, Zm), and therefore contain three-dimensional information. In this embodiment, the first line images RL are arranged at equal intervals of an angle α along the circumferential direction of a circle centered on Zm. The second line images CL are arranged at equal intervals of a distance ΔR along the radial direction of the circle centered on Zm.

[0085] After generating the first line image RL and the second line image CL, the processing device 51 converts these into images from the viewpoint of the imaging device 19 to generate a reference image 65. By converting the first line image RL and the second line image CL into images from the viewpoint of the imaging device 19, the display device 52 can deform the reference image 65 to match the shape of the work object WA and display it.

[0086] Next, in step S108, the processing device 51 removes the aforementioned occupation area SA from the generated guide image 60 and reference image 65. Because the guide image 60 and reference image 65 are generated from information about the shape of the work object WA, removing the occupation area SA from the guide image 60 and reference image 65 means removing the occupation area SA from the information about the shape of the work object WA. The processing device 51 may generate the guide image 60 and reference image 65 using information about the shape of the work object WA from which the occupation area SA has been removed.

[0087] In step S108, the processing device 51 converts the occupied area SA into an image from the viewpoint of the imaging device 19 and removes it from the guide image 60 and the reference image 65. The processing device 51 may remove the occupied area SA before being converted into an image from the viewpoint of the imaging device 19 from the first image IMC, the first line image 62, and the second line image 63 before being converted into an image from the viewpoint of the imaging device 19, and from the first line image RL and the second line image CL before being converted into an image from the viewpoint of the imaging device 19, respectively.

[0088] 15 is a diagram showing a work image 69. In step S109, the processing device 51 generates the work image 69 by combining the guide image 60 from which the occupied area SA has been removed, the reference image 65, and a second image 68 of the work object WA captured by the imaging device 19. In step S110, the processing device 51 displays the generated work image 69 on the display device 52. The work image 69 is an image obtained by combining the image 68 of the work object WA, the reference image 65, and the guide image 60.

[0089] The reference image 65 is a plurality of line images extending on the surface WAS of the work object WA along the circumferential and radial directions of a circle centered on the central axis of rotation of the upper rotating body 3, and therefore the operator of the hydraulic excavator 1 can grasp the position of the work object WA by referring to the reference image 65. For example, the operator can grasp the depth, i.e., the position in the front-to-rear direction of the upper rotating body 3 provided on the hydraulic excavator 1, by using the second line image CL, and can grasp the position of the upper rotating body 3 in the rotation direction Rtd by using the first line image RL.

[0090] Since the distance Rbt changes depending on the distance between the bucket 8 and the Zm axis, the first image IMC also moves away from or towards the hydraulic excavator 1 depending on the distance between the bucket 8 and the Zm axis. Since the first image IMC moves depending on the current position of the bucket 8, the operator can grasp the positions of the work object WA and the bucket 8 by operating the work implement 2 while referring to the first image IMC. As a result, a decrease in work efficiency is suppressed. Also, a decrease in work accuracy is suppressed.

[0091] By referring to the first image IMC and the second line image CL, the operator can easily understand whether the position of the excavation target is farther or closer to the bucket 8 when the upper rotating body 3 rotates while maintaining the current posture of the work implement 2. As a result, a decrease in work efficiency is suppressed. Also, a decrease in work accuracy is suppressed.

[0092] The first linear image 62 and the second linear image 63 indicate the position of the cutting edge 8T of the bucket 8 on the surface WAS of the work object WA. In other words, the portion of the surface WAS of the work object WA sandwiched between the first linear image 62 and the second linear image 63 is the position of the cutting edge 8T of the bucket 8. Therefore, the operator can more easily grasp the positional relationship between the bucket 8 and the work object WA using the first linear image 62 and the second linear image 63. As a result, a decrease in work efficiency is suppressed. Also, a decrease in work accuracy is suppressed.

[0093] The reference image 65 and the first image IMC are displayed along the shape of the terrain on which the work object WA, for example, the hydraulic excavator 1 is working, so the relative positional relationship between the reference image 65 and the first image IMC can be easily grasped on the terrain surface displayed in two dimensions on the display device 52. Furthermore, the first line image RL and the second line image CL that make up the reference image 65 are arranged at equal intervals in the vehicle body coordinate system, so the operator can easily get a sense of distance on the terrain surface, making it easy to grasp the sense of perspective.

[0094] In an embodiment, work image 69 may include information 64 indicating the distance between cutting edge 8T of bucket 8 and work object WA. Information 64 has the advantage of allowing the operator to grasp the actual distance between cutting edge 8T of bucket 8 and work object WA. The distance between cutting edge 8T of bucket 8 and work object WA can be the distance from cutting edge 8T at the center of bucket 8 in the width direction Wb to the surface WAS of work object WA.

[0095] Information 64 may be spatial position information related to the work tool or work object WA. Instead of or in addition to the distance between the cutting edge 8T of bucket 8 and the work object WA, the spatial position information includes information related to the posture, such as the angle of bucket 8, information indicating the relative distance between bucket 8 and work object WA, information indicating the relationship between the orientation of, for example, the cutting edge 8T of bucket 8 and the orientation of the face of work object WA, information indicating the position of bucket 8 in coordinates, information indicating the orientation of the face of work object WA, and information indicating the distance in the Xm direction in the vehicle body coordinate system from image capture device 19 to the cutting edge 8T of bucket 8.

[0096] That is, the processing device 51 may determine at least one of the position of the bucket 8, which is the work tool, the attitude of the bucket 8, the position of the work object WA, the relative attitude of the work object WA, the relative distance between the bucket 8 and the work object WA, and the relative attitude between the bucket 8 and the work object WA, and display it on the display device 52.

[0097] <Reference Image According to Modification> 16 is a diagram showing a reference image 65a according to a modified example. Unlike the reference image 65 in which the vehicle body coordinate system is a polar coordinate system, the reference image 65a is a lattice image formed of a plurality of first linear images 66 and a plurality of second linear images 67 perpendicular to the first linear images 66 when viewed from the direction of the Zm axis in the vehicle body coordinate system (Xm, Ym, Zm). The plurality of first linear images 66 are parallel to the Xm axis of the vehicle body coordinate system, and the plurality of second linear images 67 are parallel to the Ym axis of the vehicle body coordinate system. The plurality of first linear images 66 are arranged at equal intervals. The plurality of second linear images 67 are also arranged at equal intervals. The intervals at which the plurality of first linear images 66 are arranged are equal to the intervals at which the plurality of second linear images 67 are arranged.

[0098] The processing device 51 generates a work image 69 by combining the guide image 60 from which the occupied area SA has been removed, the reference image 65a, and a second image 68 of the work object WA captured by the imaging device 19. By having the processing device 51 display such a grid image on the display device 52 together with the guide image 60, the operator can easily grasp the positional relationship between the bucket 8 and the work object WA during swinging and excavation.

[0099] <Third image> FIG. 17 shows an example of the first image IMC and the third image 61. The third image 61 is a line image connecting positions Pt1 and Pt2 and extending along the surface WAS of the work object WA. Position Pt1 is the position where a line LV1 corresponding to the first line image 62 intersects with the surface WAS of the work object WA. Position Pt2 is the position where a line LV2 corresponding to the second line image 63 intersects with the surface WAS of the work object WA. The processing device 51 defines the third image 61 as a set of positions of the surface WAS of the work object WA when a line connecting the first position Pt1 and the second position Pt2 is projected onto the surface WAS of the work object WA. Position Pt, where a line extending from the cutting edge 8T at the center in the width direction Wb of the bucket 8 and parallel to the Zm axis of the vehicle coordinate system intersects with the surface WAS of the work object WA, is part of the third image 61. Position Pt is also part of the first image IMC.

[0100] The processing device 51 may display a third image 61 on the display device 52 in addition to the first image IMC. The third image 61 allows the operator to grasp the current position of the bucket 8 on the surface WAS of the work object WA. As a result, the operator can easily and reliably grasp the positional relationship between the bucket 8 and the work position. When displaying the first image IMC and the third image 61 on the display device 52, the processing device 51 may differentiate the display forms of the first image IMC and the third image 61. Differentiating the display forms includes, for example, differentiating the color of the first image IMC from the color of the third image 61, differentiating the line thickness of the first image IMC from the line thickness of the third image 61, differentiating the line type of the first image IMC from the line type of the third image 61, and a combination of two or more of these. Differentiating the display forms of the first image IMC and the third image 61 makes it easier for the operator to distinguish the third image 61 from the first image IMC.

[0101] <Control System of Hydraulic Excavator 1 According to Modified Example> Figure 18 is a diagram showing a control system 1Sa of a hydraulic excavator 1 according to a modified example. The image display system 100 and remote operation system 101 described above remotely operate the hydraulic excavator 1 using the operation device 53 of the facility 50 shown in Figure 1. In this modified example, a display device 52a is provided in the operator's cab 4 shown in Figure 2. In this modified example, a work image 69 is displayed on the display device 52a to allow the hydraulic excavator 1 to assist the operator in the work.

[0102] In the control system 1Sa, a processing device 51 and an operation device 53a are connected to the signal line 35 of the control system 1S described above. A display device 52a is connected to the processing device 51. The processing device 51 provided in the control system 1Sa has the same functions as the processing device 51 provided in the facility 50 shown in FIG. 1 in the image display system 100 and the remote operation system 101 described above. The processing device 51 and the display device 52a constitute an image display system 100a according to a modified example.

[0103] In a modified example, the processing device 51 may determine the cutting edge position P4 using the detection values ​​of the attitude detection device 32 and the sensor controller 26, or the sensor controller 26 may determine the cutting edge position P4 using the detection value of the attitude detection device 32. In the former case, the position detection device 32, the sensor controller 26, and the processing device 51 correspond to a position detection unit that detects the position of the work tool. In the latter case, the position detection device 32 and the sensor controller 26 correspond to a position detection unit that detects the position of the work tool. The position detection unit may be a distance measurement device such as a stereo camera or a laser scanner, as described above.

[0104] The display device 52a of the control system 1Sa is a head-up display that projects an image onto the windshield FG of the driver's seat. The display device 52a may be a head-mounted display. The display device 52a displays a work image 69 at a position aligned with the bucket 8 visible through the windshield FG. In this case, since the work object WA and the bucket 8 are visible through the windshield FG, the work image 69 displayed by the display device 52a only needs to include at least the first image IMC. In other words, when the display device 52a is a head-up display or a head-mounted display, the processing device 51 causes the display device 52a to display the first image IMC. In a modified example, in addition to the first image IMC, the reference image 65 is also displayed on the display device 52a.

[0105] The display device 52a is not limited to a head-up display and may be a normal display. When the display device 52a is a normal display, the processing device 51 causes the display device 52a to display a working image 69 obtained by combining at least the first image IMC and a second image 68 captured by the imaging device 19.

[0106] The operating device 53a is a device for operating the hydraulic excavator 1, and includes a left operating lever 53La and a right operating lever 53Ra. The operating device 53a may be of a pilot hydraulic type or an electric type.

[0107] In a hydraulic excavator 1 equipped with a control system 1Sa, the image display system 100a displays a work image 69 including at least the first image IMC on the windshield FG when the work object WA is viewed from a predetermined position, for example, a position corresponding to the viewpoint of an operator seated in the driver's seat. This processing allows the operator to easily and reliably grasp the positional relationship between the bucket 8 and the work position from the first image IMC displayed on the windshield FG. As a result, the image display system 100a can prevent a decrease in work efficiency and work accuracy. The farther the bucket 8 is from the vehicle body 1B of the hydraulic excavator, the more difficult it becomes for the operator to grasp the sense of perspective. In such cases, the image display system 100a can prevent a decrease in work efficiency and work accuracy.

[0108] Furthermore, by using the hydraulic excavator 1 equipped with the image display system 100a, even an inexperienced operator can easily grasp the perspective of the bucket 8. As a result, a decrease in work efficiency is suppressed. Also, a decrease in work accuracy is suppressed. Furthermore, even in a situation where it is difficult for the operator to visually check the actual work target WA, such as during night work, the operator can work while viewing the work image 69 displayed on the display device 52a, thereby suppressing a decrease in work efficiency and work accuracy.

[0109] The image display systems 100, 100a and the remote operation system 101 superimpose the guide image 60 and the reference image 65 or 65a generated from the viewpoint of the imaging device 19 on the image 68 of the actual work object WA captured by the imaging device 19 and display the superimposed image on the display device 52. Through this processing, the image display system 100 and the remote operation system 101 can make it easier for the operator remotely operating the hydraulic excavator 1 to understand the positional relationship between the position of the bucket 8 and the work object WA, using the image of the work object WA displayed on the display device 52. As a result, a decrease in work efficiency is suppressed. Also, a decrease in work accuracy is suppressed.

[0110] Since the first image IMC included in the guide image 60 extends along the rotation direction of the upper rotating body 3, the operator can easily understand the relative positional relationship between the bucket 8 and the work point in the work object WA when the upper rotating body 3 rotates by referring to the guide image 60. As a result, the image display system 100 and the remote operation system 101 can suppress a decrease in work efficiency and a decrease in work accuracy in a work machine having a rotating body.

[0111] By using the image display systems 100, 100a and the remote operation system 101, even an inexperienced operator can easily grasp the positional relationship between the position of the bucket 8 and the work point at the work object WA. As a result, a decrease in work efficiency and work accuracy is suppressed. Furthermore, the image display system 100 and the remote operation system 101 superimpose the guide image 60, the reference image 65 or 65a, and the image 68 of the actual work object WA on the display device 52, thereby allowing the operator to focus on only one screen during work, suppressing a decrease in work efficiency.

[0112] In the reference image 65 or 65a, the intervals between adjacent first line images RL are equal, and the intervals between adjacent second line images CL are equal. Therefore, by superimposing the reference image 65 or 65a on the image 68 of the actual work object WA captured by the imaging device 19, the work point at the work object WA can be easily grasped. Furthermore, by superimposing the first image IMC of the guide image 60 on the reference image 65, the operator can easily grasp the distance traveled by the bucket 8, thereby suppressing a decrease in work efficiency.

[0113] Since the occupied area SA, which is the area of ​​the work machine 2, is removed from the guide image 60 and the reference image 65 or 65a, the guide image 60 and the reference image 65 or 65a can be prevented from being distorted by the occupied area SA and from being superimposed on the work machine 2. As a result, the image display system 100 and the remote operation system 101 can display the work image 69 on the display device 52 in a format that is easy for the operator to see.

[0114] In the embodiment and the modified example, the guide image 60 only needs to include at least the first image IMC. That is, the guide image 60 does not need to include the first linear image 62 and the second linear image 63. Furthermore, the processing device 51 may change the display form of, for example, the first image IMC in the guide image 60 depending on the distance between the cutting edge 8T of the bucket 8 and the work object WA. This makes it easier for the operator to grasp the position of the bucket 8 and the distance between the work object WA.

[0115] In the embodiment, the processing device 51 only needs to display at least the first image IMC and the second image 68 on the display device 52. Even in this case, the operator can easily grasp the relative positional relationship between the bucket 8 and the work point on the work object WA when the upper rotating body 3 has rotated by referring to the first image IMC extending along the rotation direction Rtd of the upper rotating body 3. As a result, by displaying at least the first image IMC and the second image 68 on the display device 52, a decrease in work efficiency and a decrease in work accuracy are suppressed in a work machine having a rotating body.

[0116] In the embodiment, the processing device 51 generates the work image 69 in the vehicle body coordinate system (Xm, Ym, Zm), but the work image 69 may be generated in any of the global coordinate system (Xg, Yg, Zg), the imaging device coordinate system (Xc, Yc, Zc), or the coordinate system of the distance detection device 20 (Xd, Yd, Zd). When the work image 69 is generated in the global coordinate system (Xg, Yg, Zg), the GNSS antennas 21 and 22 and the position calculation device 23 are required. When the work image 69 is generated in the vehicle body coordinate system (Xm, Ym, Zm), the imaging device coordinate system (Xc, Yc, Zc), or the coordinate system of the distance detection device 20 (Xd, Yd, Zd), the GNSS antennas 21 and 22 and the position calculation device 23 are not required.

[0117] In the embodiment, the occupied area SA, which is the area of ​​the work implement 2, does not have to be removed from the guide image 60 and the reference image 65. Even in this case, the operator can easily grasp the relative positional relationship between the bucket 8 and the work point on the work object WA when the upper rotating body 3 rotates, using the first image IMC and second image 68 displayed on the display device 52. This prevents a decrease in work efficiency in a work machine with a rotating body. Furthermore, this prevents work accuracy in a work machine with a rotating body.

[0118] In the above-described embodiment, a part of the hydraulic excavator 1 detected by the distance detection device 20, for example, the bucket 8, was removed as described above, and the information on the shape of the work object WA (three-dimensional terrain data) was obtained. However, three-dimensional terrain data acquired in the past, for example, several seconds ago, may be stored in the memory unit 51M of the processing device 51, and the processing unit 51P of the processing device 51 may determine whether the current work object WA and the stored three-dimensional terrain data are in the same position. If they are in the same position, the processing device 51 may use the past three-dimensional terrain data to display the reference image 65 or 65a. In other words, even if there is terrain hidden by part of the hydraulic excavator 1 as viewed from the imaging device 19, the processing device 51 may be able to display the reference image 65 or 65a as long as there is past three-dimensional terrain data.

[0119] <Example of image display during loading work> In the embodiment and the modified example, an example of image display by the image display system 100, 100a will be described for a loading operation in which a hydraulic excavator 1 loads cargo such as rocks and earth and sand into the vessel of a dump truck. The processing device 51 of the image display system 100, 100a acquires the position of the dump truck, which is the target of the loading operation, detected by the distance detection device 20. The first image IMC and the reference image 65 or 65a included in the work image 69 are displayed at the position of the dump truck, allowing the operator to grasp the relationship between the height and position of the work implement 2 and the dump truck. By adding the position of the dump truck to the work image 69, the operator can grasp the position at which the bucket 8 of the work implement 2 will reach the dump truck when the upper rotating body 3 is rotated to move the bucket 8 of the work implement 2 closer to the dump truck. This reduces interference between the work implement 2 and the dump truck.

[0120] The processing device 51 may treat the position of the dump truck detected by the distance detection device 20 as part of the work object WA, or may acquire the position of the dump truck from a management server or the like via the communication device 25 shown in Fig. 2. In the latter case, the processing device 51 may apply information about the model of the dump truck to the acquired position of the dump truck to determine the position of the dump truck.

[0121] The processing device 51 may compare the height of the dump truck with the height of the work implement 2, specifically the height of the cutting edge 8T of the bucket 8, and change the display form of the first image IMC according to the comparison result. This processing allows the operator to understand the relationship between the height of the work implement 2 and the height of the dump truck, thereby reducing interference between the work implement 2 and the dump truck. When changing the display form of the first image, the processing device 51 can, for example, display the first image IMC in red if the height of the work implement 2 is lower than the height of the dump truck, display the first image IMC in orange if the height of the dump truck and the height of the work implement 2 are the same, and display the first image IMC in green if the height of the work implement 2 is higher than the height of the dump truck.

[0122] When at least one of the third image 61, the first straight line image 61, and the second straight line image 62 is displayed together with the first image IMC, the processing device 51 may change the display form of at least one of the third image 61, the first straight line image 61, and the second straight line image 62 together with the first image IMC, depending on the height of the dump truck. Such processing makes it easier for the operator to grasp the relationship between the height of the work implement 2 and the height of the dump truck.

[0123] In the embodiment and the modified example, as long as the work machine is provided with a revolving body having the work implement 2, the type of the work implement 2 and the type of the work machine are not limited.

[0124] Although the embodiments and modifications have been described above, the embodiments are not limited to the above content. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the scope of what is called equivalents. Furthermore, the above-described components can be combined as appropriate. Furthermore, at least one of various omissions, substitutions, and modifications of the components can be made without departing from the spirit of the embodiments and modifications. [Explanation of symbols]

[0125] 1. Hydraulic excavator 1B Vehicle body 1S, 1Sa control system 2 Work equipment 3 Upper rotating body 8 buckets 8B blade 8T cutting edge 19 Imaging device 20 Distance detection device 23 Position calculation device 50 facilities 51 Processing equipment 51IO input / output section 51M storage section 51P Processing section 52,52a Display device 60 Guide Image 61 Third image 62 First Linear Image 63 Second Linear Image 65,65a Reference image 68 2nd image 69 working images 100, 100a Image display system for work machines (image display system) 101 Remote control system for work machines (remote control system) CL Second Line Image RL First Line Image FG front window IMC 1st image Rtd Turning direction WA Work Target WAS surface Wb width direction

Claims

1. A work machine image display system that is applied to a work machine including a work implement having a work tool and a rotating body to which the work implement is attached, a processing device that acquires an image of a work target of the work machine captured by an imaging device, and displays on a display device a display image including a straight line image extending from at least one of both ends of the work tool in the width direction to the work target along a direction parallel to the central axis of rotation of the rotating body or a vertical axis in global coordinates, and a line image extending in the width direction of the work tool from a position where the straight line image intersects with the work target, Image display system for work machines.

2. The processing device displays on the display device a pair of straight line images extending from both ends of the work tool in a width direction to the work target.

2. The image display system for a work machine according to claim 1.

3. The linear image extends vertically from an end of the work tool in a width direction.

3. The image display system for a work machine according to claim 2.

4. The line image extends outside the bucket width and extends from both ends of the work tool in the width direction toward the center position. The image display system for a work machine according to any one of claims 1 to 3.

5. the processing device synthesizes the display image and the image of the work object and displays the synthesized image on the display device; The image display system for a work machine according to any one of claims 1 to 4.

6. The processing device causes the display device to display information about a transport vehicle on which a load is to be loaded by the work machine. The image display system for a work machine according to any one of claims 1 to 5.

7. The display device is provided outside the work machine. The image display system for a work machine according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Full automation construction system

    JP2004294067A

  • Display system of construction machine, and method of controlling the same

    JP2014129676A

  • Device and method for supporting shovel

    JP2016102312A

  • Shovel

    JP2016106192A