Map generation system and map generation method

Abstract

To correct a deviation generated in a case of using a scanning type distance measuring sensor.SOLUTION: A map generating system of one embodiment of this invention includes: a three-dimensional data obtaining portion obtaining three-dimensional data from a scanning type distance measuring sensor measuring a distance to an object to be measured; an operation information obtaining portion obtaining operation information representing operation of a work machine which is generated during an update period of the three-dimensional data by the distance measuring sensor; a correction amount calculating portion calculating a correction amount of each measuring point of the three-dimensional data based on the operation information; and a correction processing portion correcting the three-dimensional data by applying the correction amount to each measuring point of the three-dimensional data.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a map generation system and a map generation method. [Background technology]

[0002] When remotely operating a work machine, the image displayed from the viewpoint of the operator of the work machine is two-dimensional, so the sense of perspective is poor. Therefore, it is difficult to grasp the distance between the work object and the work machine, and work efficiency may decrease. In addition, when an operator on board the work machine operates the work machine, it may be difficult to grasp the distance between the work machine and the work object depending on the operator's level of proficiency, and work efficiency may decrease. An example of an image display system for solving such problems is described in Patent Document 1. The image display system described in Patent Document 1 includes an imaging device attached to a work machine equipped with a work machine having a work tool, an attitude detection device that detects the attitude of the work tool, a distance detection device that obtains information on the distance to the work object of the work machine, and a processing device that uses information on the position of the work tool obtained using the attitude of the work machine and information on the position of the work object obtained from the distance information obtained by the distance detection device to generate an image of a part corresponding to the work tool on the work object facing the work tool, synthesizes the image with the image of the work object captured by the imaging device, and displays it on a display device. According to the image display system described in Patent Document 1, it is possible to suppress a decrease in work efficiency when working using a work machine equipped with a work machine having a work tool. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-160741 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the image display system described in Patent Document 1, when a scanning distance sensor is used as a distance detection device that detects distance by sequentially scanning multiple measurement points (measurement points), there is a problem that if the work machine turns or moves during one scanning period, a shift in the measurement position will occur.

[0005] The present invention has been made in consideration of the above circumstances, and has an object to provide a map generation system and a map generation method that can solve the above problems. [Means for solving the problem]

[0006] In order to solve the above problems, one aspect of the present invention is a map generation system having a measurement data acquisition unit that acquires measurement data from a scanning distance sensor that measures the distance to a measured object, an operation information acquisition unit that acquires operation information representing the operation of a work machine that occurs between updates of the measurement data by the scanning distance sensor, a correction amount calculation unit that calculates a correction amount for each measurement point of the measurement data based on the operation information, and a correction processing unit that applies the correction amount to each measurement point of the measurement data to correct the measurement data. Effect of the Invention

[0007] According to an aspect of the present invention, it is possible to correct deviations that occur when using a scanning distance measuring sensor. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing a work machine image display system and a work machine remote control system according to an embodiment. [Diagram 2] FIG. 2 is a perspective view showing a hydraulic excavator which is a work machine according to the embodiment. [Diagram 3] FIG. 3 is a diagram showing a control system of a hydraulic excavator, which is a work machine according to the embodiment. [Figure 4] FIG. 4 is a block diagram showing an example of a functional configuration of the processing unit 51P shown in FIG. [Diagram 5]FIG. 5 is a diagram for explaining a coordinate system in the image display system and the remote control system according to the embodiment. [Figure 6] FIG. 6 is a rear view of the hydraulic excavator. [Figure 7] FIG. 7 is a diagram for explaining the coordinate systems of the imaging device and the distance detection device. [Figure 8] FIG. 8 is a flowchart of an example of control executed by the image display system and the remote control system. [Figure 9] FIG. 9 is a diagram for explaining the process shown in FIG. [Figure 10] FIG. 10 is a diagram for explaining the process shown in FIG. [Figure 11] FIG. 11 is a diagram for explaining the process shown in FIG. [Figure 12] FIG. 12 is a diagram for explaining the process shown in FIG. [Figure 13] FIG. 13 is a diagram for explaining the process shown in FIG. [Figure 14] FIG. 14 is a diagram showing an imaging device, a distance detection device, and a work target. [Figure 15] FIG. 15 is a diagram for explaining an occupied region. [Figure 16] FIG. 16 is a diagram showing information on the shape of the work object from which the occupied area has been removed. [Figure 17] FIG. 17 is a diagram for explaining an image showing the position of the bucket on the work target. [Figure 18] FIG. 18 is a diagram for explaining an image showing the position of the bucket on the work target. [Figure 19] FIG. 19 is a diagram for explaining an image showing the position of the bucket on the work target. [Figure 20] FIG. 20 is a diagram showing a grid image that is a reference image. [Figure 21] FIG. 21 is a diagram showing a grid image that is a reference image. [Figure 22] FIG. 22 shows a working image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings, in which the same or corresponding components are designated by the same reference numerals and their description will be omitted where appropriate.

[0010] <Outline of the work machine image display system and work machine remote control system> 1 is a diagram showing a work machine image display system 100 (map generation system) and a work machine remote operation system 101 according to the embodiment. When an operator remotely operates a hydraulic excavator 1, which is a work machine, the work machine image display system 100 (hereinafter referred to as the image display system 100) captures images of a work target of the hydraulic excavator 1, more specifically, a topographical surface that is a target of work by a work implement 2 provided on the hydraulic excavator 1, that is, the work target WA and a bucket 8, which is a work tool, with an imaging device 19, and displays the obtained images on a display device 52 (display unit). At this time, the image display system 100 displays, on the display device 52, an image 68 of the work target WA captured by the imaging device 19, a grid image 65, and an image 60 for indicating the position of the bucket 8 on the work target WA.

[0011] The image display system 100 includes an imaging device 19, a posture detection device 32, a distance detection device 20, and a processing device 51. A remote operation system 101 for a work machine (hereinafter, appropriately referred to as the remote operation system 101) includes the imaging device 19, the posture detection device 32, the distance detection device 20, a work machine control device 27, a display device 52, a processing device 51, and an operation device 53. In the embodiment, the imaging device 19, the posture detection device 32, and the distance detection device 20 of the image display system 100 are provided in the hydraulic excavator 1, and the processing device 51 is provided in a facility 50. The facility 50 is a facility for remotely operating the hydraulic excavator 1 and managing the hydraulic excavator 1. In the embodiment, the imaging device 19, the posture detection device 32, the distance detection device 20, and the work machine control device 27 of the remote operation system 101 are provided in the hydraulic excavator 1, and the display device 52, the processing device 51, and the operation device 53 are provided in the facility 50.

[0012] The processing device 51 of the image display system 100 includes a processing unit 51P, a storage unit 51M, and an input / output unit 51IO. The processing unit 51P is a processor such as a CPU (Central Processing Unit). The storage unit 51M is, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk drive, a storage device, or a combination of these. The input / output unit 51IO is an interface circuit for connecting the processing device 51 to an external device. In the embodiment, the input / output unit 51IO is connected to a display device 52, an operating device 53, and a communication device 54 as external devices. The external devices connected to the input / output unit 51IO are not limited to these.

[0013] The processing device 51 uses information on the position of the bucket 8, which is a working tool, obtained using the attitude of the working machine 2, and information on the position of the work object WA obtained from distance information determined by the distance detection device 20 to generate an image of a portion of the work object WA facing the bucket 8, which corresponds to the bucket 8, with the imaging device 19 as a reference. Then, the processing device 51 combines the generated image with the image of the work object WA captured by the imaging device 19, and displays the combined image on the display device 52. The work object WA is a surface on which the work machine 2 of the hydraulic excavator 1 will perform work such as excavation or leveling.

[0014] The display device 52 is exemplified by, but not limited to, a liquid crystal display or a projector. The communication device 54 includes an antenna 54A. The communication device 54 communicates with the communication device 25 provided in the hydraulic excavator 1 to obtain information about the hydraulic excavator 1 and transmit information to the hydraulic excavator 1.

[0015] The operating device 53 has a left operating lever 53L installed on the left side of the operator, and a right operating lever 53R arranged on the right side of the operator. The forward / backward and left / right movements of the left operating lever 53L and the right operating lever 53R correspond to two-axis movements. For example, forward / backward operation of the right operating lever 53R corresponds to operation of the boom 6 of the work implement 2 provided in the hydraulic excavator 1. Left / right operation of the right operating lever 53R corresponds to operation of the bucket 8 of the work implement 2. Forward / backward operation of the left operating lever 53L corresponds to operation of the arm 7 of the work implement 2. Left / right operation of the left operating lever 53L corresponds to rotation of the upper rotating body 3 of the hydraulic excavator 1.

[0016] The amount of operation of the left operating lever 53L and the right operating lever 53R is detected by, for example, a potentiometer and a Hall IC, and the processing device 51 generates a control signal for controlling the solenoid control valve based on these detected values. This signal is sent to the work machine control device 27 via the communication device 54 of the facility 50 and the communication device 25 of the hydraulic excavator 1. The work machine control device 27 controls the solenoid control valve based on the control signal, thereby controlling the work machine 2. The solenoid control valve will be described later.

[0017] The processing device 51 acquires an input to at least one of the left operating lever 53L and the right operating lever 53R, and generates a command for operating at least one of the work implement 2 and the upper rotating body 3. The processing device 51 transmits the generated command to the communication device 25 of the hydraulic excavator 1 via the communication device 54. The work implement control device 27 provided in the hydraulic excavator 1 acquires the command from the processing device 51 via the communication device 25, and operates at least one of the work implement 2 and the upper rotating body 3 in accordance with the command.

[0018] The hydraulic excavator 1 includes a communication device 25, a work machine control device 27, a posture detection device 32, an imaging device 19, a distance detection device 20, antennas 21 and 22, and a global position calculation device 23. The work machine control device 27 controls the work machine 2. The communication device 25 is connected to the antenna 24 and communicates with a communication device 54 provided in the facility 50. The work machine control device 27 controls the work machine 2 and the upper rotating body 3. The posture detection device 32 detects the posture of at least one of the work machine 2 and the hydraulic excavator 1. The imaging device 19 is attached to the hydraulic excavator 1 and captures an image of the work target WA. The distance detection device 20 obtains information on the distance from a predetermined position of the hydraulic excavator 1 to the work target WA. The antennas 21 and 22 receive radio waves from a positioning satellite 200. The global position calculation device 23 obtains the global positions of the antennas 21 and 22, i.e., the positions in the global coordinate system, using the radio waves received by the antennas 21 and 22.

[0019] <Overall configuration of hydraulic excavator 1> FIG. 2 is a perspective view showing a hydraulic excavator 1 which is a work machine according to an embodiment. The hydraulic excavator 1 has a vehicle body 1B as a main body and a work machine 2. The vehicle body 1B has an upper rotating body 3 which is a rotating body and a traveling device 5 which is a traveling body. The upper rotating body 3 accommodates devices such as an engine and a hydraulic pump which are a power generating device inside an engine room 3EG. In the embodiment, the hydraulic excavator 1 uses an internal combustion engine such as a diesel engine as the engine which is the power generating device, but the power generating device is not limited to an internal combustion engine. The power generating device of the hydraulic excavator 1 may be, for example, a so-called hybrid type device which combines an internal combustion engine, a generator motor, and a power storage device. The power generating device of the hydraulic excavator 1 may be a device which does not have an internal combustion engine and which combines a power storage device and a generator motor.

[0020] The upper rotating body 3 has a cab 4. The cab 4 is installed on one end side of the upper rotating body 3. In other words, the cab 4 is installed on the opposite side to the side where the engine room 3EG is located. A handrail 9 is attached to the upper part of the upper rotating body 3.

[0021] The traveling device 5 mounts the upper rotating body 3. The traveling device 5 has tracks 5a, 5b. The traveling device 5 is driven by one or both of hydraulic motors 5c provided on the left and right sides. The tracks 5a, 5b of the traveling device 5 rotate to cause the hydraulic excavator 1 to travel. The work machine 2 is attached to the lateral side of the operator's cab 4 of the upper rotating body 3.

[0022] The hydraulic excavator 1 may be equipped with tires instead of the tracks 5a, 5b, and a traveling device capable of traveling by transmitting the driving force of the engine to the tires via a transmission. An example of such a hydraulic excavator 1 is a wheeled hydraulic excavator. The hydraulic excavator 1 may also be equipped with a traveling device having such tires, and further with a working machine attached to the vehicle body (main body), and may be, for example, a backhoe loader having a structure not equipped with the upper rotating body 3 and its rotating mechanism as shown in FIG. 1. In other words, a backhoe loader is equipped with a working machine attached to the vehicle body, and is equipped with a traveling device that constitutes a part of the vehicle body.

[0023] The side of the upper rotating body 3 where the work machine 2 and the driver's cab 4 are located is the front, and the side where the engine room 3EG is located is the rear. The fore-and-aft direction of the upper rotating body 3 is the y direction. The left side as you face the front is the left of the upper rotating body 3, and the right side as you face the front is the right of the upper rotating body 3. The left-right direction of the upper rotating body 3 is also called the width direction or x direction. In the hydraulic excavator 1 or the vehicle body 1B, the traveling gear 5 side is the bottom relative to the upper rotating body 3, and the upper rotating body 3 side is the top relative to the traveling gear 5. The up-down direction of the upper rotating body 3 is the z direction. When the hydraulic excavator 1 is installed on a horizontal surface, the bottom is the vertical direction, i.e., the side in the direction of gravity, and the top is the opposite side to the vertical direction.

[0024] The work machine 2 has a boom 6, an arm 7, and a bucket 8, which is a working tool, a boom cylinder 10, an arm cylinder 11, and a bucket cylinder 12. The base end of the boom 6 is rotatably attached to the front of the vehicle body 1B via a boom pin 13. The base end of the arm 7 is rotatably attached to the tip of the boom 6 via an arm pin 14. A bucket 8 is attached to the tip of the arm 7 via a bucket pin 15. The bucket 8 rotates around the bucket pin 15. A plurality of blades 8B are attached to the bucket 8 on the side opposite the bucket pin 15. The blade tip 8T is the tip of the blade 8B.

[0025] The bucket 8 does not have to have multiple blades 8B. In other words, the bucket may not have the blade 8B as shown in FIG. 2, and the blade tip may be formed in a straight shape by a steel plate. The working machine 2 may have, for example, a tilt bucket having a single blade. The tilt bucket is a bucket that has a bucket tilt cylinder and tilts left and right so that even if the hydraulic excavator is on a slope, the bucket can be shaped into a free shape on a slope or flat ground, and the ground can be leveled, and compacted with a bottom plate. In addition, the working machine 2 may have, instead of the bucket 8, a slope bucket or a rock-cutting attachment equipped with a rock-cutting tip, as a working tool.

[0026] The boom cylinder 10, arm cylinder 11, and bucket cylinder 12 shown in Fig. 2 are hydraulic cylinders driven by the pressure of hydraulic oil discharged from a hydraulic pump. The boom cylinder 10 drives the boom 6 to rotate it. The arm cylinder 11 drives the arm 7 to rotate it around the arm pin 14. The bucket cylinder 12 drives the bucket 8 to rotate it around the bucket pin 15.

[0027] Antennas 21, 22 and an antenna 24 are attached to the upper part of the upper rotating body 3. The antennas 21, 22 are used to detect the current position of the hydraulic excavator 1. The antennas 21, 22 are electrically connected to a global position calculation device 23 shown in FIG. 3. The global position calculation device 23 is a position detection device that detects the position of the hydraulic excavator 1. The global position calculation device 23 detects the current position of the hydraulic excavator 1 by using RTK-GNSS (Real Time Kinematic - Global Navigation Satellite Systems, GNSS stands for Global Navigation Satellite System). In the following description, the antennas 21, 22 are appropriately referred to as GNSS antennas 21, 22. Signals corresponding to the GNSS radio waves received by the GNSS antennas 21, 22 are input to the global position calculation device 23. The global position calculation device 23 determines the installation positions of the GNSS antennas 21, 22 in the global coordinate system. An example of a global navigation satellite system is the Global Positioning System (GPS), but the global navigation satellite system is not limited to this.

[0028] 2, the GNSS antennas 21, 22 are preferably installed on the upper rotating body 3 at both ends separated in the left-right direction, i.e., in the width direction, of the hydraulic excavator 1. In the embodiment, the GNSS antennas 21, 22 are attached to handrails 9 attached to both sides in the width direction of the upper rotating body 3. The positions at which the GNSS antennas 21, 22 are attached to the upper rotating body 3 are not limited to the handrails 9, but it is preferable to install the GNSS antennas 21, 22 as far apart as possible, since this improves the detection accuracy of the current position of the hydraulic excavator 1. In addition, the GNSS antennas 21, 22 are preferably installed at positions that do not obstruct the operator's view as much as possible.

[0029] 1, and the distance detection device 20 determines the distance from itself (a predetermined position of the hydraulic excavator 1) to the work object WA, so it is preferable to obtain information from as wide an area of ​​the work object WA as possible. For this reason, in the embodiment, the antenna 24, the imaging device 19, and the distance detection device 20 are installed above the operator's cab 4 of the upper rotating body 3. The location where the imaging device 19 and the distance detection device 20 are installed is not limited to above the operator's seat 4. For example, the imaging device 19 and the distance detection device 20 may be installed inside and above the operator's cab 4.

[0030] The imaging device 19 has an imaging surface 19L facing forward of the upper rotating body 3. The distance detection device 20 has a detection surface 20L facing forward of the upper rotating body 3. In the embodiment, the imaging device 19 is a monocular camera equipped with an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0031] In the embodiment, the distance detection device 20 is a three-dimensional laser range finder, a three-dimensional laser scanner, a three-dimensional distance sensor, or the like. The three-dimensional laser range finder, or the like, is also called LiDAR (Light Detection and Ranging), and is a sensor (scanning distance sensor) that irradiates a laser light emitted in a pulsed manner while sequentially scanning the measurement direction in a plurality of measurement directions over a certain range, and measures the distance and direction based on the time until the reflected scattered light returns and the irradiation direction. In this embodiment, the distance detection device 20 sequentially stores, updates, and outputs three-dimensional data (measurement data) indicating the measurement results of each measurement point (each reflection point) for each scanning period. The three-dimensional data output by the distance detection device 20 is point cloud data indicating the distance and direction to each measurement point or the three-dimensional coordinate value of each measurement point. Note that LiDAR is superior in terms of long-distance and outdoor compatibility. In other embodiments, it is also possible to remove the portion of the two-dimensional scan data corresponding to the work machine as measurement data, but in this embodiment, a correction amount is calculated for the three-dimensional scan data in order to apply it to three-dimensional topographical data.

[0032] The three-dimensional laser range finder is not limited to the above-mentioned pulse-type (time of flight detection type) light wave distance meter, but may be an intensity modulation type (phase difference detection type) light wave distance meter.

[0033] <Control system of hydraulic excavator 1> 3 is a diagram showing a control system 1S of a hydraulic excavator 1 which is a work machine according to the embodiment. The control system 1S includes a communication device 25, a sensor controller 26, a work machine control device 27, an imaging device 19, a distance detection device 20, a global position calculation device 23, an attitude detection device 32, an IMU (Inertial Measurement Unit) 33, and a hydraulic system 36. The communication device 25, the sensor controller 26, and the work machine control device 27 are connected by a signal line 35. With this structure, the communication device 25, the sensor controller 26, and the work machine control device 27 can exchange information with each other via the signal line 35. An example of the signal line that transmits information within the control system 1S is an in-vehicle signal line such as a controller area network (CAN).

[0034] The sensor controller 26 has a processor such as a CPU (Central Processing Unit) and a storage device such as a RAM and a ROM. The sensor controller 26 receives the detection values ​​of the global position calculation device 23, information on the image captured by the imaging device 19, the detection values ​​of the distance detection device 20, the detection values ​​of the attitude detection device 32, and the detection values ​​of the IMU 33. The sensor controller 26 transmits the input detection values ​​and image information to a processing device 51 of the facility 50 shown in FIG. 1 via a signal line 35 and the communication device 25.

[0035] The work machine control device 27 has a processor such as a CPU, and storage devices such as a RAM (Random Access Memory) and a ROM (Read Only Memory). The work machine control device 27 acquires, via the communication device 25, commands for operating at least one of the work machine 2 and the upper rotating body 3, which have been generated by a processing device 51 of the facility 50. The work machine control device 27 controls the solenoid control valve 28 of the hydraulic system 36 based on the acquired commands.

[0036] The hydraulic system 36 includes an electromagnetic control valve 28, a hydraulic pump 29, and hydraulic actuators such as the boom cylinder 10, the arm cylinder 11, the bucket cylinder 12, and the swing motor 30. The hydraulic pump 29 is driven by an engine 31 and discharges hydraulic oil for operating the hydraulic actuators. The work implement control device 27 controls the flow rate of hydraulic oil supplied to the boom cylinder 10, the arm cylinder 11, the bucket cylinder 12, and the swing motor 30 by controlling the electromagnetic control valve 28. In this manner, the work implement control device 27 controls the operation of the boom cylinder 10, the arm cylinder 11, the bucket cylinder 12, and the swing motor 30.

[0037] The sensor controller 26 acquires detection values ​​of the first stroke sensor 16, the second stroke sensor 17, and the third stroke sensor 18. The first stroke sensor 16 is provided in the boom cylinder 10, the second stroke sensor 17 in the arm cylinder 11, and the third stroke sensor 18 in the bucket cylinder 12.

[0038] The first stroke sensor 16 detects a boom cylinder length, which is the length of the boom cylinder 10, and outputs the result to the sensor controller 26. The second stroke sensor 17 detects an arm cylinder length, which is the length of the arm cylinder 11, and outputs the result to the sensor controller 26. The third stroke sensor 18 detects a bucket cylinder length, which is the length of the bucket cylinder 12, and outputs the result to the sensor controller 26.

[0039] Once the boom cylinder length, arm cylinder length, and bucket cylinder length are determined, the posture of the work machine 2 is determined. Therefore, the first stroke sensor 16, second stroke sensor 17, and third stroke sensor 18 that detect these correspond to a posture detection device 32 that detects the posture of the work machine 2. The posture detection device 32 is not limited to the first stroke sensor 16, second stroke sensor 17, and third stroke sensor 18, and may be an angle detector.

[0040] The sensor controller 26 calculates the inclination angle of the boom 6 in a direction (z-axis direction) perpendicular to the horizontal plane in the local coordinate system, which is the coordinate system of the hydraulic excavator 1, from the boom cylinder length detected by the first stroke sensor 16. The work machine control device 27 calculates the inclination angle of the arm 7 relative to the boom 6 from the arm cylinder length detected by the second stroke sensor 17. The work machine control device 27 calculates the inclination angle of the bucket 8 relative to the arm 7 from the bucket cylinder length detected by the third stroke sensor 18. The inclination angles of the boom 6, arm 7, and bucket 8 are information indicating the attitude of the work machine 2. That is, the sensor controller 26 obtains information indicating the attitude of the work machine 2. The sensor controller 26 transmits the calculated inclination angle to a processing device 51 of the facility 50 shown in FIG. 1 via a signal line 35 and the communication device 25.

[0041] The GNSS antenna 21 receives a position P1 indicating its own position from a positioning satellite. The GNSS antenna 22 receives a position P2 indicating its own position from the positioning satellite. The GNSS antennas 21 and 22 receive the positions P1 and P2 at a cycle of, for example, 10 Hz. The positions P1 and P2 are information on the positions where the GNSS antennas are installed in the global coordinate system. Signals corresponding to the GNSS radio waves received by the GNSS antennas 21 and 22, i.e., the positions P1 and P2, are input to the global position calculation device 23. The GNSS antennas 21 and 22 output the positions P1 and P2 to the global position calculation device 23 every time they receive them.

[0042] The global position calculation device 23 has a processor such as a CPU and a storage device such as a RAM and a ROM. The global position calculation device 23 detects positions P1 and P2 of the GNSS antennas 21 and 22 in the global coordinate system at a frequency of, for example, 10 Hz, and outputs the positions P1 and P2 as reference position information Pga1 and Pga2 to the sensor controller 26. In the embodiment, the global position calculation device 23 determines the azimuth angle of the hydraulic excavator 1, more specifically, the yaw angle which is the azimuth angle of the upper rotating body 3, from the two acquired positions P1 and P2, and outputs the determined azimuth angle to the sensor controller 26. The sensor controller 26 transmits the acquired reference position information Pga1 and Pga2 and the yaw angle to a processing device 51 of a facility 50 shown in FIG. 1 via a signal line 35 and a communication device 25.

[0043] The IMU 33 detects the motion and attitude of the hydraulic excavator 1. The motion of the hydraulic excavator 1 includes at least one of the motion of the upper rotating body 3 and the motion of the traveling gear 5. The attitude of the hydraulic excavator 1 can be represented by the roll angle, pitch angle, and yaw angle of the hydraulic excavator 1. In the embodiment, the IMU 33 detects and outputs the angular velocity and acceleration of the hydraulic excavator 1.

[0044] <Example of Functional Configuration of Processing Unit 51P> FIG. 4 is a block diagram showing an example of a functional configuration of the processing unit 51P shown in FIG. 1. As shown in FIG. 4, the processing unit 51P of this embodiment has, as functional elements formed by a combination of hardware and software, a three-dimensional data acquisition unit 51P1 (measurement data acquisition unit), an operation information acquisition unit 51P2, a correction amount calculation unit 51P3, a correction processing unit 51P4, and a display processing unit 51P5. The three-dimensional data acquisition unit 51P1 acquires three-dimensional data from a distance detection device 20 (scanning distance measurement sensor) that updates three-dimensional data measured at a predetermined period, via a sensor controller 26 or the like. The operation information acquisition unit 51P2 acquires operation information representing the operation of the hydraulic excavator 1 (working machine) that occurs during the period of updating the three-dimensional data by the distance detection device 20, via the sensor controller 26 or the like. Here, the operation information includes information indicating the position of the hydraulic excavator 1 and the attitude of the work machine 2. The correction amount calculation unit 51P3 calculates a correction amount for each measurement point of the three-dimensional data based on the operation information. The correction processing unit 51P4 applies the correction amount to each measurement point of the three-dimensional data to correct the three-dimensional data. Then, the display processing unit 51P5 superimposes an image based on the corrected three-dimensional data on an image captured by the imaging device 19 (camera) and displays it on the display device 52. This configuration makes it possible to correct deviations that occur when using a distance detection device 20 consisting of a scanning distance measuring sensor. However, the processing unit 51P may have, for example, a three-dimensional data acquisition unit 51P1, a motion information acquisition unit 51P2, a correction amount calculation unit 51P3, and a correction processing unit 51P4 without having the display processing unit 51P5, and perform processing to correct the three-dimensional data and generate a three-dimensional map.

[0045] The correction processing unit 51P4 converts the three-dimensional data from the coordinate system of the distance detection device 20 (scanning distance measurement sensor) to the coordinate system of the hydraulic excavator 1 (work machine), corrects the three-dimensional data by applying a correction amount to each measurement point of the three-dimensional data, and converts the corrected three-dimensional data to a global coordinate system. This configuration makes it possible to correct deviations caused by movements such as turning and translation (translation). This point will be described later.

[0046] <About the coordinate system> FIG. 5 is a diagram for explaining the coordinate system in the image display system 100 and the remote control system 101 according to the embodiment. FIG. 6 is a rear view of the hydraulic excavator 1. FIG. 7 is a diagram for explaining the coordinate system of the imaging device and the distance detection device. In the image display system 100 and the remote control system 101, there are a global coordinate system, a local coordinate system, a coordinate system of the imaging device 19, and a coordinate system of the distance detection device 20 (hereinafter also referred to as a sensor coordinate system). In the embodiment, the global coordinate system is, for example, a coordinate system in GNSS. The global coordinate system is a three-dimensional coordinate system represented by (X, Y, Z) based on a reference position PG of, for example, a reference pile 80, which is a reference installed in the work section GA of the hydraulic excavator 1. As shown in FIG. 5, the reference position PG is located at, for example, the tip 80T of the reference pile 80 installed in the work section GA.

[0047] The local coordinate system is a three-dimensional coordinate system represented by (x, y, z) with the hydraulic excavator 1 as the reference. In the embodiment, the origin position PL of the local coordinate system is the intersection point of the z axis, which is the central axis of rotation of the upper rotating body 3, and a plane perpendicular to the z axis within the swing circle of the upper rotating body 3, but is not limited to this. The plane perpendicular to the z axis within the swing circle can be a plane passing through the center of the swing circle in the z-axis direction.

[0048] In the embodiment, the coordinate system of the imaging device 19 is a three-dimensional coordinate system represented by (Xc, Yc, Zc) with the center of the light receiving surface 19P of the imaging element 19RC as the origin PC, as shown in Fig. 7. In the embodiment, the coordinate system of the distance detection device 20 is a three-dimensional coordinate system represented by (Xd, Yd, Zd) with the center of the light receiving surface 20P of the distance detection element 20RC as the origin PD, as shown in Fig. 7.

[0049] <Posture of hydraulic excavator 1> As shown in FIG. 6, the inclination angle θ4 of the upper rotating body 3 in the left-right direction, i.e., in the width direction, is the roll angle of the hydraulic excavator 1, and as shown in FIG. 5, the inclination angle θ5 of the upper rotating body 3 in the front-rear direction is the pitch angle of the hydraulic excavator 1, and the angle of the upper rotating body 3 around the z-axis is the yaw angle of the hydraulic excavator 1. The roll angle is obtained by integrating the angular velocity around the y-axis detected by the IMU 33 over time, the pitch angle is obtained by integrating the angular velocity around the x-axis detected by the IMU 33 over time, and the yaw angle is obtained by integrating the angular velocity around the z-axis detected by the IMU 33 over time. The angular velocity around the z-axis is the swing angular velocity ω of the hydraulic excavator 1. That is, the yaw angle of the hydraulic excavator 1, more specifically, of the upper rotating body 3, is obtained by integrating the swing angular velocity ω over time.

[0050] The acceleration and angular velocity detected by the IMU 33 are output to the sensor controller 26 as operation information. The sensor controller 26 performs processing such as filtering and integration on the operation information acquired from the IMU 33 to obtain the inclination angle θ4 which is the roll angle, the inclination angle θ5 which is the pitch angle, and the yaw angle. The sensor controller 26 transmits the obtained inclination angle θ4, inclination angle θ5, and yaw angle as information related to the attitude of the hydraulic excavator 1 to a processing device 51 of a facility 50 shown in Fig. 1 via a signal line 35 and a communication device 25 shown in Fig. 3.

[0051] As described above, the sensor controller 26 obtains information indicating the attitude of the work machine 2. Specifically, the information indicating the attitude of the work machine 2 is the inclination angle θ1 of the boom 6 in the direction (z-axis direction) perpendicular to the horizontal plane in the local coordinate system, the inclination angle θ2 of the arm 7 relative to the boom 6, and the inclination angle θ3 of the bucket 8 relative to the arm 7. A processing device 51 of a facility 50 shown in Fig. 1 calculates a position P4 of a cutting edge 8T of the bucket 8 (hereinafter referred to as cutting edge position as appropriate) from the information indicating the attitude of the work machine 2 acquired from the sensor controller 26 of the hydraulic excavator 1, i.e., the inclination angles θ1, θ2, and θ3.

[0052] The memory unit 51M of the processing device 51 stores data of the working machine 2 (hereinafter, appropriately referred to as working machine data). The working machine data includes the length L1 of the boom 6, the length L2 of the arm 7, and the length L3 of the bucket 8. As shown in FIG. 5, the length L1 of the boom 6 corresponds to the length from the boom pin 13 to the arm pin 14. The length L2 of the arm 7 corresponds to the length from the arm pin 14 to the bucket pin 15. The length L3 of the bucket 8 corresponds to the length from the bucket pin 15 to the blade tip 8T of the bucket 8. The blade tip 8T is the tip of the blade 8B shown in FIG. 2. The working machine data also includes information on the position up to the boom pin 13 relative to the origin position PL of the local coordinate system. The processing device 51 can obtain the blade tip position P4 relative to the origin position PL using the lengths L1, L2, L3, the inclination angles θ1, θ2, θ3, and the origin position PL. In the embodiment, the processing device 51 of the facility 50 determines the cutting edge position P4, but the sensor controller 26 of the hydraulic excavator 1 may determine the cutting edge position P4 and transmit it to the processing device 51 of the facility 50.

[0053] <Example of control executed by the image display system 100 and the remote control system 101> Fig. 8 is a flowchart of an example of control executed by the image display system 100 and the remote operation system 101. Fig. 14 is a diagram showing the image capture device 19, the distance detection device 20, and the work object WA.

[0054] In step S101, the sensor controller 26 shown in FIG. 3 acquires information about the hydraulic excavator 1. The information about the hydraulic excavator 1 is information obtained from the imaging device 19, the distance detection device 20, the global position calculation device 23, the attitude detection device 32, and the IMU 33. As shown in FIG. 14, the imaging device 19 captures an image of the work object WA within an imaging range TA to obtain an image of the work object WA. The distance detection device 20 detects a distance Ld from the distance detection device 20 to the work object WA and other objects present within a detection range MA. The global position calculation device 23 obtains reference position information Pga1, Pga2 corresponding to positions P1, P2 of the GNSS antennas 21, 22 in the global coordinate system. The attitude detection device 32 detects the boom cylinder length, the arm cylinder length, and the bucket cylinder length. The IMU 33 detects the attitude of the hydraulic excavator 1, more specifically, the roll angle θ4, the pitch angle θ5, and the yaw angle of the upper rotating body 3.

[0055] In step S102, the image display system 100 and the processing device 51 of the remote operation system 101 acquire information of the hydraulic excavator 1 from the sensor controller 26 of the hydraulic excavator 1 via the communication device 25 of the hydraulic excavator 1 and the communication device 54 connected to the processing device 51, by the three-dimensional data acquisition unit 51P1 or the operation information acquisition unit 51P2.

[0056] The information of the hydraulic excavator 1 that the processing device 51 acquires from the sensor controller 26 includes an image of the work object WA captured by the imaging device 19, information on the distance from the distance detection device 20 to the work object WA detected by the distance detection device 20, information on the attitude of the work implement 2 equipped on the hydraulic excavator 1 detected by the attitude detection device 32, reference position information Pga1, Pga2, and information on the attitude of the hydraulic excavator 1.

[0057] The information on the distance from the distance detection device 20 to the work object WA includes the distance Ld to the work object WA or object OB present within the detection range MA, and the information on the orientation of the position Pd corresponding to the distance Ld. In FIG. 14, the distance Ld is shown as the distance to the work object WA. The information on the orientation of the position Pd is the orientation of the position Pd when the distance detection device 20 is used as a reference, and is the angle with respect to each axis Xd, Yd, Zd of the coordinate system of the distance detection device 20. The information on the attitude of the work machine 2 acquired by the processing device 51 is the inclination angles θ1, θ2, θ3 of the work machine 2 calculated by the sensor controller 26 using the boom cylinder length, the arm cylinder length, and the bucket cylinder length. The information on the attitude of the hydraulic excavator 1 is the roll angle θ4, pitch angle θ5, and yaw angle of the hydraulic excavator 1, more specifically, the upper rotating body 3.

[0058] Processing device 51, for example, by correction amount calculation unit 51P3, uses tilt angles θ1, θ2, θ3 of work machine 2 acquired from sensor controller 26 and length L1 of boom 6, length L2 of arm 7, and length L3 of bucket 8 stored in memory unit 51M to determine blade tip position P4 of bucket 8. Blade tip position P4 of bucket 8 is a set of coordinates in the local coordinate system (x, y, z) of hydraulic excavator 1.

[0059] Proceeding to step S103, the processing device 51 determines, by the correction processing unit 51P4, the correction amounts (translational movement amount and rotation amount) from the information of the hydraulic excavator 1 at the start and end of measurement by the distance detection device 20. How to determine the correction amounts will be described later.

[0060] Proceeding to step S104, the processing device 51 converts the distance Ld to the work object WA into position information using information on the distance to the work object WA, for example, by the correction amount calculation unit 51P3 or the correction processing unit 51P4. The position information is the coordinates of the position Pd in ​​the coordinate system (Xd, Yd, Zd) of the distance detection device 20. In step S104, all distances Ld detected by the distance detection device 20 within the detection range MA are converted into position information. The processing device 51 converts the distance Ld into position information using, for example, the correction amount calculation unit 51P3 or the correction processing unit 51P4, using the distance Ld and information on the direction of the position Pd corresponding to the distance Ld. In step S104, the distance to the object OB present within the detection range MA is also converted into position information, similar to the distance Ld to the work object WA. By the processing of step S104, information on the position of the work object WA within the detection range MA can be obtained. Information on the shape of the work object WA can be obtained from the information on the position of the work object WA.

[0061] The position information and shape information of the work object WA are a set of coordinates of a position Pd in ​​the coordinate system (Xd, Yd, Zd) of the distance detection device 20. The processing device 51 converts the shape information of the work object WA into values ​​in the coordinate system (Xc, Yc, Zc) of the imaging device 19, and then converts it into values ​​in the local coordinate system (x, y, z) of the hydraulic excavator 1, for example, by using the correction amount calculation unit 51P3 or the correction processing unit 51P4.

[0062] Proceeding to step S105, the processing device 51 corrects the local coordinate values ​​of each measurement point by the correction processing unit 51P4 based on the correction amount calculated in step S103. Here, the processing of step S103 and the processing of step S105 will be described with reference to Figs. 9 to 13. Figs. 9 to 13 are diagrams for explaining the processing of step S103 and the processing of step S105 shown in Fig. 8. Note that the conversion equation described later is a general equation that also corresponds to rotation and translational movement, rotation about a central axis other than the z-axis passing through the origin position PL, and the like, but in Figs. 9 to 13, the correction processing in this embodiment will be described using only the rotation of the hydraulic excavator 1 (rotation about the z-axis passing through the origin position PL) as an example of displacement.

[0063] When the distance detection device 20 is a scanning distance measuring sensor, as shown in Figure 9, if the hydraulic excavator 1 rotates while the distance detection device 20 is measuring one scanning period (also called one frame), when the position measured at the beginning of one scanning period is converted to a position in the global coordinate system based on the local coordinate system at the end of one scanning period, the position of the initial measurement point will have a larger position error than the measurement point measured near the end.

[0064] Furthermore, as shown in FIG. 10, when multiple three-dimensional data are measured for a certain measurement object OB1, when a map (three-dimensional map) of the measurement object OB1 is generated at the angle at which the measurement is completed, the map is generated with the measurement object OB1 distorted in the rotation direction.

[0065] In this embodiment, in the local coordinate system shown in FIG. 11, in order to correct the position (x, y, z) that has been shifted and converted by the rotation shown in FIG. 12 to the actually measured position (x', y', z'), the rotation angle θz calculated from the difference between the angle of one frame before and the angle of the current frame is divided by the number of measurement points N per frame, Δθ=θz / N, which is set as the angle of shift for each measurement point. Then, as shown in FIG. 13, in the Lissajous scanning of the measurement points, the correction angle Δθi of the i-th measurement point is set to Δθi=Δθ×(Ni). For example, the correction angle Δθ1 of the first measurement point is Δθ1=Δθ×(N-1), the correction angle Δθ2 of the second measurement point is Δθ2=Δθ×(N-2), and the correction angle Δθ3 of the third measurement point is Δθ3=Δθ×(N-3). In addition, the correction angle ΔθN-1 of the (N-1)th measurement point is ΔθN-1=Δθ, and the correction angle ΔθN of the Nth measurement point is ΔθN=0.

[0066] In addition, the actually measured position (x', y', z') can be calculated from the shifted and converted position (x, y, z) and the correction angle Δθi of the measurement point using the following formula.

[0067]

number

[0068] In addition, the measurement point coordinate P S The coordinates of the measurement point in the global coordinate system P G The transformation procedure for converting to the above consists of the following steps (1) to (3). That is, (1) first, the sensor coordinate system is converted to a local coordinate system (vehicle body coordinate system). (2) Next, the measurement error during turning (including movement and oscillation) is corrected using a homogeneous transformation matrix. (3) Then, the local coordinate system is converted to a global coordinate system. However, while the above procedure performs the correction from the local coordinate system, it is also possible to convert to global coordinates first and then perform the correction.

[0069] According to the above steps (1) to (3), the measurement point coordinate P in the sensor coordinate system is calculated in response to the rotation, translation, and general swing of the hydraulic excavator 1. S The coordinates of the measurement point in the global coordinate system P G The conversion formula for converting to is as follows:

[0070]

number

[0071] Here, R E’toG is a homogeneous transformation matrix (4x4 matrix) from the local coordinate system to the global coordinate system, and is applied to the position at the end of the scan. R EtoE’ is a homogeneous correction transformation matrix (4x4 matrix), which is a transformation matrix for correcting the deviation caused by the movement of the hydraulic excavator 1 (work machine) during one scan, and is a matrix that represents the difference from the position at the time of the i-th measurement point to the position at the end of the scan (different for each i-th point). R StoE is a homogeneous transformation matrix (4×4 matrix) from the sensor coordinate system to the local coordinate system, and is a matrix that is uniquely determined by the mounting position of the distance detection device 20 on the hydraulic excavator 1 (vehicle body).

[0072] In addition, the homogeneous transformation matrix R EtoE’ (4×4 matrix) with 3×1 matrix T EtoE’ is expressed by the following equation.

[0073]

number

[0074] Here, ΔP is calculated as follows: ΔP=(local coordinate origin position at the end of measurement for one frame by the distance detection device 20)−(local coordinate origin position at the time of measurement of the measurement point in question).

[0075] In addition, the homogeneous transformation matrix R EtoE’ (4×4 matrix) with 3×3 matrix R EtoE’ is the product of the roll, pitch, and yaw rotation matrices, and is expressed by the following equation.

[0076]

number

[0077] Here, RΔyaw, RΔpitch, RΔroll=(attitude angle of the hydraulic excavator 1 at the end of measurement for one frame by the distance detection device 20)−(attitude angle of the hydraulic excavator 1 at the time of measurement at the measurement point in question).

[0078] In addition, the coordinates of the (i-th) measurement point in the global coordinate system P G Let X' = (X', Y', Z', 1) (a 4 × 1 matrix), and the (i-th) measurement point coordinate P S When Xd=(xd, yd, zd, 1) (4×1 matrix) is used and translational movement is not taken into consideration (translation matrix T is a zero matrix), the conversion formula is as follows. Here, the coordinate values ​​(xd, yd, zd) are the initial point cloud data obtained from distance detection device 20. Furthermore, the coordinate values ​​(X', Y', Z') are the final values ​​to be found.

[0079]

number

[0080] Here, the matrix T is a parallel matrix with a 3×1 determinant. dtoxis a transformation matrix (4×4 matrix) from the sensor coordinate system to the local coordinate system (vehicle body coordinate system), and is a matrix that is uniquely determined by the mounting position of the distance detection device 20 on the hydraulic excavator 1 (vehicle body). R xtox' is a transformation matrix (4x4 matrix) for correcting the deviation caused by the movement of the hydraulic excavator 1 (work machine) during one scan, and is a matrix that represents the difference from the position at the time of the i-th measurement point to the position at the end of the scan (different for each i-th point). x'toX' is the (corrected) local to global transformation matrix (4x4 determinant) applied to the end-of-scan position.

[0081] In addition, the 4×4 matrix R dtox , R xtox' and R x'toX is a 3×3 matrix R, which considers only rotation when there is no translation. dtox , R xtox' and R x'toX can be substituted into the following equation using

[0082]

number

[0083] Here, A1 to A4 represent the following formulas.

[0084]

number

[0085] In the above formula, only the yaw angle is taken into consideration, and it is expressed as a rotation matrix of the yaw angle. yaw is the rotation angle of the upper rotating body 3. The rotation matrices in the pitch, roll and yaw directions are as follows:

[0086]

number

[0087] In step S106, the processing device 51 converts the information on the position of the work object WA, the blade tip position P4 of the bucket 8, and the reference position information Pga1, Pga2 acquired from the sensor controller 26 of the hydraulic excavator 1 into a global coordinate system (X, Y, Z) by the display processing device 51P5. In converting into the global coordinate system (X, Y, Z), the processing device 51 generates a rotation matrix using the roll angle θ4, pitch angle θ5, and yaw angle of the hydraulic excavator 1 acquired from the sensor controller 26 by the display processing device 51P5. The processing device 51 converts the information on the position of the work object WA, the blade tip position P4 of the bucket 8, and the reference position information Pga1, Pga2 into the global coordinate system (X, Y, Z) by using the generated rotation matrix by the display processing device 51P5. Next, the process proceeds to step S107, and the processing device 51 obtains an occupied area by the display processing device 51P5.

[0088] FIG. 15 is a diagram illustrating the occupation area SA. The occupation area SA is an area occupied by the work machine 2 within the information on the shape of the work object WA. In the example shown in FIG. 15, a part of the bucket 8 of the work machine 2 is within the detection range MA of the distance detection device 20, and is between the distance detection device 20 and the work object WA. For this reason, the distance detection device 20 detects the distance to the bucket 8, not the distance to the work object WA, for the portion of the occupation area SA. In the embodiment, the processing device 51 removes the portion of the occupation area SA from the information on the shape of the work object WA obtained in step S104 by the display processing unit 51P5.

[0089] The processing device 51 stores, for example, in the storage unit 51M, information on at least one of the position and posture detected by the distance detection device 20 according to at least one of the position and posture of the bucket 8 by the display processing unit 51P5. In this embodiment, such information is included in the posture of the working machine 2 of the hydraulic excavator 1. The posture of the working machine 2 can be obtained by using the inclination angles θ1, θ2, and θ3 of the working machine 2, the length L1 of the boom 6, the length L2 of the arm 7, and the length L3 of the bucket 8, and by using the posture of the hydraulic excavator 1 as necessary. Then, the processing device 51 compares the data detected by the distance detection device 20 with the information stored in the storage unit 51M by the display processing unit 51P5, and if the two match, it can be determined that the bucket 8 has been detected. By performing such processing using the posture of the working machine 2, the processing device 51 does not use information on the bucket 8 of the occupation area SA when generating the grid image 65 shown in FIG. 1, so that the grid image 65 can be generated accurately.

[0090] Note that processing using the attitude of the work machine 2 to remove portions of the occupation area SA may be performed by the following method. Information relating to at least one of the position and attitude of the bucket 8 in the global coordinate system, which is included in the attitude of the work machine 2, is obtained from the tilt angles θ1, θ2, θ3 of the work machine 2, the length L1 of the boom 6, the length L2 of the arm 7, and the length L3 of the bucket 8. In steps S104 and S106, information on the shape of the work object WA in the global coordinate system is obtained. In step S108, the processing device 51 uses the display processing unit 51P5 to remove an area obtained by projecting the position of the bucket 8 onto the information on the shape of the work object WA as the occupation area SA from the shape of the work object WA.

[0091] FIG. 16 is a diagram showing information on the shape of the work object WA with the occupied area removed. Information IMWA on the shape of the work object WA is a set of coordinates Pgd(X,Y,Z) in the global coordinate system (X,Y,Z). As a result of the processing of step S108, no coordinate information exists for the occupied area IMBA. Next, proceeding to step S109, the processing device 51 generates an image showing the position of the bucket 8 by the display processing unit 51P4. The image showing the position of the bucket 8 is an image of the portion of the work object WA that corresponds to the bucket 8.

[0092] 17 to 19 are diagrams for explaining an image showing the position of the bucket 8 on the work object WA. In the embodiment, the image showing the position of the bucket 8 is an image showing the position of the cutting edge 8T of the bucket 8 on the work object WA. Hereinafter, the image showing the position of the cutting edge 8T of the bucket 8 will be appropriately referred to as a cutting edge position image. As shown in FIG. 17, the cutting edge position image is an image defined by the position Pgt(X, Y, Z) of the surface WAP of the work object WA when the cutting edge 8T is projected onto the work object WA in the vertical direction, that is, in the direction in which gravity acts. The vertical direction is the Z direction in the global coordinate system (X, Y, Z), and is perpendicular to the X and Y directions.

[0093] As shown in FIG. 18, a line image formed along the surface WAP of the work object WA between the first position Pgt1 (X1, Y1, Z1) and the second position Pgt2 (X2, Y2, Z2) of the surface WAP of the work object WA is the cutting edge position image 61. The first position Pgt1 (X1, Y1, Z1) is an intersection point between a straight line LV1 extending vertically from an outer position Pgb1 of the blade 8B at one end 8Wt1 side of the width direction Wb of the bucket 8 and the surface WAP of the work object WA. The second position Pgt2 (X2, Y2, Z2) is an intersection point between a straight line LV2 extending vertically from an outer position Pgb2 of the blade 8B at the other end 8Wt2 side of the width direction Wb of the bucket 8 and the surface WAP of the work object WA. The width direction Wb of the bucket 8 is the direction in which the multiple blades 8B are arranged.

[0094] The processing device 51 uses the display processing unit 51P4 to determine straight lines LV1 and LV2 extending vertically from positions Pgb1 and Pgb2 of the bucket 8. Next, the processing device 51 uses the display processing unit 51P4 to determine a first position Pgt1 (X1, Y1, Z1) and a second position Pgt2 (X2, Y2, Z2) from the obtained straight lines LV1 and LV2 and information on the shape of the work object WA. Then, the processing device 51 uses the display processing unit 51P4 to project a straight line connecting the first position Pgt1 and the second position Pgt2 onto the surface WAP of the work object WA, and sets the set of positions Pgt of the surface WAP as a cutting edge position image 61.

[0095] In the embodiment, the processing device 51 generates, by the display processing unit 51P4, a first line image 62 which is an image of a line LV1 connecting the position Pgb1 and a first position Pgt1 (X1, Y1, Z1), and a second line image 63 which is an image of a line LV2 connecting the position Pgb2 and a second position Pgt2 (X2, Y2, Z2). Next, the processing device 51 converts, by the display processing unit 51P4, the cutting edge position image 61, the first line image 62, and the second line image 63 into images based on the imaging device 19, i.e., images from the viewpoint of the imaging device 19.

[0096] 19, the image at the viewpoint of the imaging device 19 is an image obtained when a blade-tip position image 61, a first line image 62, and a second line image 63 are viewed from an origin Pgc (Xc, Yc, Zc) of the imaging device in the global coordinate system (X, Y, Z). The origin Pgc (Xc, Yc, Zc) of the imaging device is the coordinate obtained by converting the center of the light receiving surface 19P of the imaging element 19RC included in the imaging device 19, i.e., the origin PC, into the global coordinate system (X, Y, Z).

[0097] The cutting edge position image 61, the first straight line image 62, and the second straight line image 63 are images in a three-dimensional space, but the image from the viewpoint of the imaging device 19 is a two-dimensional image. Therefore, the processing device 51 executes perspective projection transformation by the display processing unit 51P4 to project the cutting edge position image 61, the first straight line image 62, and the second straight line image 63 defined in the three-dimensional space, i.e., the global coordinate system (X, Y, Z), onto a two-dimensional surface. The cutting edge position image 61, the first straight line image 62, and the second straight line image 63 converted into the images from the viewpoint of the imaging device 19 will be referred to as a tool guide image 60 below as appropriate.

[0098] 20 and 21 are diagrams showing a lattice image 65 which is a reference image. After the work tool guide image 60 is generated, the process proceeds to step S110, where the processing device 51 generates a lattice image 65 which is a reference image by the display processing unit 51P4. The lattice image 65 is a line image along the surface WAP of the work object WA using information on the position of the work object WA. The lattice image 65 is a lattice including a plurality of first line images 66 and a plurality of second line images 67 intersecting with the plurality of first line images 66. In the embodiment, the first line image 66 is, for example, a line image which extends parallel to the X direction in the global coordinate system (X, Y, Z) and is arranged in the Y direction. The first line image 66 may be a line image which extends parallel to the front-rear direction of the upper rotating body 3 of the hydraulic excavator 1 in the global coordinate system (X, Y, Z) and is arranged in the width direction of the upper rotating body 3.

[0099] The grid image 65 is generated using information on the position of the work object WA, more specifically, the position Pgg (X, Y, Z) of the surface WAP. The intersection of the first line image 66 and the second line image 67 is the position Pgg (X, Y, Z). As shown in FIG. 21, the first line image 66 and the second line image 67 are defined in the global coordinate system (X, Y, Z), and therefore contain three-dimensional information. In the embodiment, the multiple first line images 66 are arranged at equal intervals, and the multiple second line images 67 are arranged at equal intervals. The interval between adjacent first line images 66 and the interval between adjacent second line images 67 are equal.

[0100] The grid image 65 is an image obtained by converting a first line image 66 and a second line image 67 generated using the position Pgg (X, Y, Z) of the surface WAP into an image from the viewpoint of the imaging device 19. After generating the first line image 66 and the second line image 67 by the display processing unit 51P4, the processing device 51 converts these into images from the viewpoint of the imaging device 19 to generate the grid image 65. By converting the first line image 66 and the second line image 67 into images from the viewpoint of the imaging device 19, the grid image 65, which is equally spaced in the horizontal plane, can be displayed after being deformed to match the shape of the work object WA in order to assist in the absolute distance of the work object WA.

[0101] Next, in step S111, the processing device 51 removes the above-mentioned occupation area SA from the generated work tool guide image 60 and the grid image 65, which is the reference image, by the display processing unit 51P4. In step S111, the processing device 51 converts the occupation area SA into an image of the viewpoint of the imaging device 19 by the display processing unit 51P4, and removes it from the work tool guide image 60 and the grid image 65, which is the reference image. In the embodiment, the processing device 51 may remove the occupation area SA before being converted into the image of the viewpoint of the imaging device 19 from the cutting edge position image 61, the first straight line image 62, and the second straight line image 63 before being converted into the image of the viewpoint of the imaging device 19, and the first line image 66 and the second line image 67 before being converted into the image of the viewpoint of the imaging device 19, respectively, by the display processing unit 51P4.

[0102] 22 is a diagram showing a work image 69. In step S112, the processing device 51 generates a work image 69 by synthesizing the work tool guide image 60 from which the occupation area SA has been removed, the lattice image 65, and the image 68 of the work object WA captured by the imaging device 19, using the display processing unit 51P4. In step S113, the processing device 51 displays the work image 68 generated by the display device 52 using the display processing unit 51P4. The work image 69 is an image in which the lattice image 65 and the work tool guide image 60 are displayed on the image 68 of the work object WA.

[0103] Since the lattice image 65 is a lattice that follows the surface WAP of the work object WA, the operator of the hydraulic excavator 1 can grasp the position of the work object WA by referring to the lattice image 65. For example, the operator can grasp the depth, that is, the front-to-rear position of the upper rotating body 3 provided on the hydraulic excavator 1, from the second line image 67, and can grasp the position of the bucket 8 in the width direction from the first line image 66.

[0104] In the work tool guide image 60, the cutting edge position image 61 is displayed along the surface WAP of the work object WA and the grid image 65. In the example shown in FIG. 22, the extension line image 61-1 and the extension line image 61-2, which are extensions of the cutting edge position image 61, are also displayed along the surface WAP of the work object WA and the grid image 65. Therefore, the operator can grasp the positional relationship between the bucket 8 and the work object WA from the grid image 65 and the cutting edge position image 61, thereby improving work efficiency and work accuracy. In the embodiment, the first straight line image 62 and the second straight line image 63 connect both ends of the cutting edge position image 61 from both sides of the width direction Wb of the bucket 8. The operator can more easily grasp the positional relationship between the bucket 8 and the work object WA from the first straight line image 62 and the second straight line image 63. The grid image 65 and the cutting edge position image 61 are displayed along the shape of the terrain (work object WA) that is the work object, so that the relative positional relationship between the two on the terrain surface (two-dimensionally) can be easily grasped. Furthermore, the first line image 66 and the second line image 67 that make up the grid image 65 are arranged at equal intervals in the global coordinate system, making it easy to grasp the sense of distance on the topographical surface and to understand the sense of perspective.

[0105] In the embodiment, the work image 69 includes information 64 indicating the distance between the cutting edge 8T of the bucket 8 and the work object WA. This has the advantage that the operator can grasp the actual distance between the cutting edge 8T of the bucket 8 and the work object WA. The distance between the cutting edge 8T of the bucket 8 and the work object WA can be the distance from the cutting edge 8T at the center of the bucket 8 in the width direction Wb to the surface WAP of the work object WA.

[0106] In addition, the information 64 may be any spatial position information related to the work tool or work object W, including, instead of or in addition to the distance between the cutting edge 8T of the bucket 8 and the work object WA, information related to the posture, such as the angle of the bucket 8, information indicating the relative distance between the bucket 8 and the work object WA, information indicating the relationship between the orientation of, for example, the cutting edge 8T of the bucket 8 and the orientation of the surface of the work object WA, information indicating the position of the bucket 8 in coordinates, information indicating the orientation of the surface of the work object WA, and information indicating the distance in the y direction in the local coordinate system from the imaging device 19 to the cutting edge 8T of the bucket 8.

[0107] In other words, the processing device 51 may determine, via the display processing unit 51P4, at least one of the position of the bucket 8, which is a work tool, the posture of the bucket 8, the position of the work object WA, the relative posture of the work object WA, the relative distance between the bucket 8 and the work object WA, and the relative posture between the bucket 8 and the work object WA, and display it on the display device 52.

[0108] As described above, the image display system 100 and the remote operation system 101 superimpose the tool guide image 60 and the grid image 65 generated from the viewpoint of the imaging device 19 on the image 68 of the actual work object WA captured by the imaging device 19 and display them on the display device 52. Through such processing, the image display system 100 and the remote operation system 101 can make it easier for the operator remotely operating the hydraulic excavator 1 to understand the positional relationship between the position of the bucket 8 and the work object WA using the image of the work object WA displayed on the display device 52, thereby improving work efficiency and work accuracy. Even an inexperienced operator can easily understand the positional relationship between the position of the bucket 8 and the work object WA by using the image display system 100 and the remote operation system 101. As a result, a decrease in work efficiency and work accuracy is suppressed. Furthermore, the image display system 100 and the remote operation system 101 superimpose the tool guide image 60, the grid image 65, and the image 68 of the actual work object WA and display them on the display device 52, thereby making it possible to improve work efficiency by making the operator focus on a single screen during work.

[0109] In the lattice image 65, the interval between adjacent first line images 66 is equal to the interval between adjacent second line images 67. For this reason, by superimposing and displaying the lattice image 65 and an image 68 of the actual work object WA captured by the imaging device 19, it becomes easier to grasp the work point at the work object WA. In addition, by superimposing the cutting edge position image 61 of the work tool guide image 60 and the lattice image 65, the operator can easily grasp the distance that the bucket 8 has moved, thereby improving work efficiency.

[0110] Since the occupation area SA, which is the area of ​​the work machine 2, is removed from the work tool guide image 60 and the lattice image 65, it is possible to prevent the work tool guide image 60 and the lattice image 65 from being distorted by the occupation area SA and from being superimposed on the work machine 2. As a result, the image display system 100 and the remote operation system 101 can display the work image 69 on the display device 52 in a form that is easy for the operator to see.

[0111] In the embodiment, the work tool guide image 60 may include at least the cutting edge position image 61. The lattice image 65 may include at least a plurality of second line images 67, i.e., a plurality of line images indicating a direction perpendicular to the front-rear direction of the upper rotating body 3 included in the hydraulic excavator 1. Furthermore, the processing device 51 may change the color of, for example, the cutting edge position image 61 in the work tool guide image 60 according to the distance between the cutting edge 8T of the bucket 8 and the work object WA. This makes it easier for the operator to grasp the distance between the position of the bucket 8 and the work object WA.

[0112] In the embodiment, the processing device 51 converts the shape information of the work object WA into the global coordinate system (X, Y, Z) to generate the work tool guide image 60 and the lattice image 65, but the shape information of the work object WA does not have to be converted into the global coordinate system (X, Y, Z). In this case, the processing device 51 handles the shape information of the work object WA in the local coordinate system (x, y, z) of the hydraulic excavator 1 and generates the work tool guide image 60 and the lattice image 65. When the shape information of the work object WA is handled in the local coordinate system (x, y, z) of the hydraulic excavator 1, the GNSS antennas 21, 22 and the global position calculation device 23 are not necessary.

[0113] In the above-described embodiment, a part of the hydraulic excavator 1 detected by the distance detection device 20 (for example, the bucket 8 as described above) is removed, and the information on the shape of the work object WA (three-dimensional terrain data) is obtained. However, three-dimensional terrain data acquired in the past (for example, several seconds ago) may be stored in the memory unit 51M of the processing device 51, and the processing unit 51P of the processing device 51 may determine whether the current work object WA and the stored three-dimensional terrain data are in the same position, and if they are in the same position, the past three-dimensional terrain data may be used to display the grid image 65. In other words, even if there is a terrain hidden by a part of the hydraulic excavator 1 as viewed from the imaging device 19, the processing device 51 may display the grid image 65 as long as there is past three-dimensional terrain data.

[0114] Also, instead of displaying the grid image 65 using a grid, for example, the grid image 65 may be displayed using a polar coordinate system instead of a local coordinate system. Specifically, concentric circles at equal intervals according to the distance from the center of the hydraulic excavator 1 (for example, the rotation center of the upper rotating body 3) may be drawn as a line image (second line image), and radial line images (first line image) may be drawn at equal intervals from the rotation center according to the rotation angle of the upper rotating body 3. In this case, the second line image, which is a line image of concentric circles, intersects with the first line image, which is a line image radiating from the rotation center. By displaying such a grid image, the positional relationship between the bucket 8 and the work object WA during rotation or excavation can be easily grasped.

[0115] <Modification of the control system of the hydraulic excavator 1> The image display system 100 and the remote operation system 101 described above remotely operate the hydraulic excavator 1 using the operation device 53 of the facility 50 shown in FIG. 1. However, the display device 52 may be provided in the cab 4 shown in FIG. 2, or an image 69 for work may be displayed on the display device 52 in the cab 4 to assist the operator in the work of the hydraulic excavator 1. In this case, the hydraulic excavator 1 can make the operator operating the hydraulic excavator 1 easily understand the positional relationship between the position of the bucket 8 and the work object WA by using the image of the work object WA displayed on the display device 52. As a result, the work efficiency and the accuracy of the work can be improved. In addition, even an inexperienced operator can easily understand the positional relationship between the position of the bucket 8 and the work object WA. As a result, the decrease in the work efficiency and the accuracy of the work is suppressed. Furthermore, in the case of night work, even in a situation where it is difficult for the operator to visually see the actual work object WA, the operator can work while looking at the work tool guide image 60 and the grid image 65 displayed on the display device 52, so that the decrease in the work efficiency is suppressed.

[0116] Although the embodiments have been described above, the embodiments are not limited to the above contents. The above-described components include those that a person skilled in the art can easily imagine, those that are substantially the same, and those that are within the scope of what is called equivalent. Furthermore, the above-described components can be combined as appropriate. Furthermore, at least one of various omissions, substitutions, and modifications of the components can be made without departing from the gist of the embodiments. The work machine is not limited to the hydraulic excavator 1, and may be other work machines such as a wheel loader or a bulldozer. [Explanation of symbols]

[0117] 1 hydraulic excavator, 1B vehicle body, 1S control system, 2 work machine, 3 upper rotating body, 4 driver's seat, 6 boom, 7 arm, 8 bucket, 8B blade, 8T blade tip, 16 first stroke sensor, 17 second stroke sensor, 18 third stroke sensor, 19 imaging device, 20 distance detection device, 21, 22 antenna (GNSS antenna), 23 global position calculation device, 26 sensor controller, 27 work machine control device, 32 attitude detection device, 33 IMU, 50 facility, 51 processing device, 52 display device, 53 operation device, 60 work tool guide image (image), 61 blade tip position image, 62 first straight line image, 63 second straight line image, 65 grid image, 66 first line image, 67 second line image, 68 image, 69 work image, 100 work machine image display system (map generation system), 101 Remote control system for work machines (remote control system)

Claims

1. a measurement data acquisition unit that acquires measurement data of a distance to a measurement target measured using a scanning distance measuring sensor provided in the work machine; an operation information acquisition unit that acquires operation information representing an operation of the work machine that has occurred during an update period of the measurement data by the scanning distance measuring sensor; a correction amount calculation unit that calculates a correction amount for each measurement point of the measurement data based on the operation information; a correction processing unit that applies the correction amount to each measurement point of the measurement data to correct the measurement data; having The operation information includes at least information indicating a rotation angle of an upper rotating body of the work machine. Map generation system.

2. The correction processing unit converts the measurement data from the coordinate system of the scanning distance measuring sensor to the coordinate system of the work machine, corrects the measurement data by applying the correction amount to each measurement point of the measurement data, and converts the corrected measurement data into a global coordinate system. The map generating system of claim 1 .

3. The image capturing apparatus further includes a display processing unit that displays an image based on the corrected measurement data by superimposing the image on the image captured by the imaging device. A map generating system according to claim 1 or 2.

4. A step of acquiring measurement data of a distance to a measured object measured using a scanning distance measuring sensor provided on the work machine; acquiring operation information representing an operation of the work machine occurring during an update period of the measurement data by the scanning distance measuring sensor; calculating a correction amount for each measurement point of the measurement data based on the operation information; applying the correction amount to each measurement point of the measurement data to correct the measurement data; Including, The operation information includes at least information indicating a rotation angle of an upper rotating body of the work machine. Map generation method.

Images