Calculation device, calculation method, and program

Abstract

To enable the acquisition of three-dimensional information obtained from a viewpoint on a heavy machine.SOLUTION: A computing device receives measurement data measured by tracking a position of a reflection prism 102 on a heavy machine 100 equipped with a laser scanner 101 and the reflection prism 102 using a total station 200 of which a position and attitude on a specific coordinate system are known, acquires a laser scan point group obtained by the laser scanner 101, transforms coordinates of the laser scan point group to those in the specific coordinate system on the basis of the measurement data and the attitude of the laser scanner 200, detects a change on a time axis in point group data obtained through the coordinate transformation, and identifies contents of work of the heavy machine on the basis of the change on the time axis of the point group data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a technique for obtaining three-dimensional data using heavy machinery.

Background Art

[0002] A technique for measuring the position of a heavy machine using a surveying device is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to digitize the results of the work performed by a heavy machine in one day or a specific period, conventionally, after the work by the heavy machine is completed, three-dimensional information of the work target is newly obtained using a surveying device such as a laser scanner installed on site. This method involves separate operations for the work by the heavy machine and the digitization of the work results, resulting in poor efficiency and a desire for improvement. In addition, the above work requires a person with specialized knowledge, and in this regard as well, efficiency improvement has been sought. Against such a background, the present invention aims to improve the efficiency of the work related to digitizing the results of the work performed by a heavy machine.

Means for Solving the Problems

[0005] The present invention is a computing device comprising: a measurement data receiving unit that receives measurement data while tracking the position of the reflective prism of a heavy machine equipped with a laser scanner and a reflective prism using a surveying device whose position and orientation on a specific coordinate system are known; a laser scan point cloud acquisition unit that acquires a laser scan point cloud obtained by the laser scanner; a coordinate transformation unit that transforms the laser scan point cloud to the specific coordinate system based on the measurement data and the orientation of the laser scanner; a change detection unit that detects changes in the point cloud data on the time axis obtained by the coordinate transformation; and a work content identification unit that identifies the content of the work of the heavy machine based on the changes in the point cloud data on the time axis.

[0006] One embodiment of the present invention is that which includes a monitoring target detection unit that detects an object moving on the ground toward the heavy machinery based on the changes in the point cloud data on the time axis.

[0007] In the present invention, one embodiment includes a notification unit that performs notification processing when the monitored object approaches the heavy machinery to a predetermined distance or less. In the present invention, one embodiment includes a removal unit that removes the point cloud of the laser scan, which includes a point cloud of a part of the heavy machinery.

[0008] The present invention is a calculation method that receives measurement data while tracking the position of the reflective prism of a heavy machine equipped with a laser scanner and a reflective prism using a surveying device whose position and orientation on a specific coordinate system are known, acquires a laser scan point cloud obtained by the laser scanner, transforms the laser scan point cloud to the specific coordinate system based on the measurement data and the orientation of the laser scanner, detects changes in the point cloud data on the time axis obtained by the coordinate transformation, and identifies the content of the work of the heavy machine based on the changes in the point cloud data on the time axis.

[0009] The present invention is a program that is read and executed by a computer, and the computer operates as follows: a measurement data receiving unit that receives measurement data measured while tracking the position of the reflective prism of a heavy machine equipped with a laser scanner and a reflective prism using a surveying device whose position and orientation on a specific coordinate system are known; a laser scan point cloud acquisition unit that acquires a laser scan point cloud obtained by the laser scanner; a coordinate transformation unit that transforms the laser scan point cloud to the specific coordinate system based on the measurement data and the orientation of the laser scanner; a change detection unit that detects changes in the point cloud data on the time axis obtained by the coordinate transformation; and a work content identification unit that identifies the content of the work of the heavy machine based on the changes in the point cloud data on the time axis.

[0010] The present invention is a computing device comprising: a positioning data receiving unit that receives positioning data from a GNSS position measuring device mounted on heavy machinery; a laser scan point cloud acquisition unit that acquires a laser scan point cloud obtained by the laser scanner; a coordinate transformation unit that transforms the laser scan point cloud into an absolute coordinate system based on the positioning data and the attitude of the laser scanner; a change detection unit that detects changes in the point cloud data on the time axis obtained by the coordinate transformation; and a work content identification unit that identifies the content of the work of the heavy machinery based on the changes in the point cloud data on the time axis. [Effects of the Invention]

[0011] The process of digitizing the results of work performed by heavy machinery will be made more efficient. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram of an embodiment. [Figure 2] This diagram shows the process of civil engineering work using heavy machinery. [Figure 3] This figure shows the changes in terrain resulting from civil engineering work using heavy machinery. [Figure 4] This is a block diagram of the arithmetic unit. [Figure 5]This is a flowchart showing an example of the processing procedure. [Figure 6] This is a flowchart showing an example of the processing procedure. [Modes for carrying out the invention]

[0013] 1. First Embodiment (overview) Figure 1 shows a heavy construction machine 100 performing civil engineering work and a total station 200 measuring the position of the heavy construction machine 100. While performing work, the heavy construction machine 100 performs a laser scan using its mounted laser scanner (Lidar) 101, acquiring a laser scan point cloud around the machine. During this process, the total station 200 repeatedly and continuously measures the position of a reflective prism 102 mounted on the heavy construction machine 100. Based on the measured position of the reflective prism 102, the laser scan point cloud is transformed onto a specific coordinate system, obtaining point cloud data that describes the position of each point on that specific coordinate system. Furthermore, changes in the point cloud data obtained through the coordinate transformation over time are detected, and based on these changes, the nature of the work performed by the heavy construction machine 100 is identified.

[0014] (Heavy machinery) Heavy equipment 100 is a power shovel. A power shovel is just one example; the type of heavy equipment is not particularly limited as long as it is used for civil engineering work. Heavy equipment 100 has a base unit 120 that travels on the ground on tracks and a rotating unit 110 that rotates horizontally on the base unit 120. The rotating unit 110 has a driver's seat and an arm 151, and a bucket 152 is positioned at the end of the arm 151. These structures are the same as those of a normal power shovel.

[0015] On the upper part of the rotating part 110, a laser scanner 101, a reflection prism 102, and a camera 103 are arranged. The position and orientation of the laser scanner 101 in the heavy machine 100 (rotating part 110), and the positional relationship between the laser scanner 101 and the reflection prism 102 are acquired in advance and regarded as known information. The laser scanner 101 scans laser scan light point by point and measures the distance and direction of each reflection point to obtain a laser scan point cloud. The laser scanner 101 is directed forward of the rotating part 110 and is arranged to laser scan the object of the work performed by the heavy machine 100.

[0016] The laser scanner 101 is provided with time information from the GNSS device 111, and the position data of each point of the laser scan point cloud measured by the laser scanner 101 is obtained in association with the time information at the time of measurement.

[0017] The form of the laser scanner 101 is not particularly limited. As the laser scanner 101, there are forms such as having two rotating parts whose rotation axes are orthogonal, one rotating part having an optical system for emitting and incident scanning light, performing laser scanning by emitting pulsed scanning light while rotating both rotating parts, performing laser scanning by reciprocating the optical system left and right and further swinging it up and down, and performing scanning electronically rather than mechanically.

[0018] The range of the laser scan is set so that the working range of the heavy machine 100 is included. In this example, the model of the laser scanner 101 and the scan range are set so that the movement range of the bucket 152 is included in the laser scan range.

[0019] A form using a plurality of laser scanners is also possible. For example, a form of expanding the scan range using a plurality of laser scanners is possible.

[0020] The reflecting prism 102 is an optical reflection target used for surveying using laser light. Here, a total reflection prism is used as the reflecting prism 102. The reflecting prism 102 reflects the incident light by changing its direction by 180°. As the optical reflection target, in addition to the reflecting prism, a reflection target having retroreflective characteristics can be used.

[0021] The camera 103 is a digital still camera that continuously and repeatedly takes still images or a camera that takes videos. The camera 103 is arranged so that the shooting range overlaps with the scan range of the laser scanner 101. FIG. 1 shows an example of arranging one camera 101 on the heavy machine 100, but it is also possible to arrange a plurality of cameras directed in multiple directions. Also, a stereo camera can be used. A depth camera can also be used as the camera.

[0022] When associating the captured image of the camera 103 with the point cloud data obtained by the laser scanner 101, 3D data based on the captured image can also be obtained. In this case, 3D data of the shooting object is obtained from the captured image using the principle of SFM (Structure from Motion). The position and orientation of the camera 103 on the heavy machine 100 may be known or unknown. When using SFM, the position and orientation of the camera at the time of shooting are calculated by adjustment calculation, so the position and orientation of the camera 103 on the heavy machine 100 may be unknown. This technology is described in, for example, Japanese Patent Application No. 2022-147113. Of course, the position and orientation of the camera 103 on the heavy machine 100 may be known.

[0023] The heavy machine 100 includes a GNSS device 111, an IMU 112, a travel detection device 113, a rotation detection device 114, and an arithmetic device 300. Although the description is omitted, the heavy machine 100 has other functions necessary for the operation of the power shovel.

[0024] The GNSS device 111 is a device that measures position using GNSS (Global Navigation Satellite System). The GNSS device 111 can be of normal accuracy and does not need to be a high-precision device using relative positioning. The positional relationship between the laser scanner 101, the reflective prism 102, the GNSS device 111, and the IMU 112 in the heavy machinery 100 is known.

[0025] IMU112 is an inertial measurement device that measures acceleration and detects changes in attitude. IMU112 is calibrated using positioning data from GNSS device 111 and outputs measured attitude values ​​in absolute coordinate system. Absolute coordinate system is the coordinate system used in GNSS and maps. For example, suppose heavy machinery 100 moves in a straight line for a small distance. In this case, its direction of movement in absolute coordinate system is calculated from the positioning data of GNSS device 111, and the attitude of IMU112 in absolute coordinate system is determined. By performing this process continuously, the attitude of IMU112 in absolute coordinate system and any changes thereto can be detected.

[0026] The IMU 112 receives time information from the GNSS device 111, and the attitude measurement data is output in association with the time information at the time of measurement.

[0027] Alternatively, the position of the reflecting prism 102 can be measured using the total station 200 to detect the movement of the IMU 112, and the IMU 112 can be calibrated based on this. In this case, the IMU 112 can be calibrated on the coordinate system from which the position and orientation of the total station 200 were determined, and the orientation of the IMU 112 and any changes in it within that coordinate system can be detected.

[0028] The GNSS device 111 is equipped with a high-precision clock, and the time information obtained from it is provided to the laser scanner 101, IMU 112, and camera 103.

[0029] The travel detection device 113 detects whether the base unit 120 is traveling on the tracks. When the heavy machinery 100 is moving, the travel detection device 113 outputs a signal indicating that the heavy machinery 100 is moving (the base unit 120 is traveling on the tracks). This signal allows for the determination of whether the heavy machinery 100 is moving (traveling). When the rotating unit 110 is rotating, the rotation detection unit 114 outputs a signal indicating that the rotating unit 110 is rotating. This signal allows for the determination of whether the rotating unit 110 is rotating. The calculation unit 300 will be described later.

[0030] The position of the heavy machinery 100 is determined by the position of the IMU 112. The position of the heavy machinery 100 can also be determined by the position of the reflecting prism 102, a point on the rotational axis of the rotating part 110, or any other position on the heavy machinery 100.

[0031] (T-Station) The Total Station 200 is an example of a surveying device capable of measuring position. The Total Station 200 features a laser-based positioning function, a camera, a clock, a data storage device for surveyed data, a communication interface, a user interface, a function to locate the survey target (reflector prism 102), and a function to track the survey target even if it moves. The Total Station 200 can utilize commonly available models.

[0032] Prior to processing, the position and orientation of the total station 200 in a specific coordinate system are acquired and treated as known data. The specific coordinate system used is either an absolute coordinate system or a local coordinate system. If an absolute coordinate system is used to determine the position of IMU112, the position and orientation of the total station 200 are acquired in the absolute coordinate system. If a local coordinate system is used to determine the position of IMU112, the position and orientation of the total station 200 are acquired in the local coordinate system. The position of the total station 200 is determined by the position of the optical origin of the optical system used for distance measurement.

[0033] The operation of the heavy machinery 100 begins with the total station 200 sighting and locking onto the reflecting prism 102. While the heavy machinery 100 is operating, the total station 200 tracks the reflecting prism 102 and repeatedly measures its position. The interval between these repeated measurements of the reflecting prism 102's position is approximately 0.1 to 5 seconds.

[0034] The Total Station 200 is equipped with a clock and acquires time data during positioning, associating it with positioning data.

[0035] (calculation section) The calculation unit 300 performs calculations and other calculations related to the laser scan point cloud acquired by the laser scanner 101. The calculation unit 300 is a computer and is equipped with a CPU, storage device, and various interfaces.

[0036] Figure 4 shows a block diagram of the calculation unit 300. The calculation unit 300 includes a point cloud data acquisition unit 301, an unnecessary data removal unit 302, a positioning data acquisition unit 303, an operating status acquisition unit 304, an attitude data acquisition unit 306, a coordinate transformation unit 307, a coordinate-transformed point cloud data acquisition unit 308, a change detection unit 309, a 3D model creation unit 310, a data storage unit 311, a work content identification unit 312, a monitoring target identification unit 313, and a notification unit 314.

[0037] Some or all of these functional units are realized by the execution of an operating program by the CPU of the arithmetic unit 300. Some or all of these functional units may be configured with dedicated hardware (electronic circuits).

[0038] The point cloud data acquisition unit 301 acquires the laser scan point cloud obtained by the laser scan performed by the laser scanner 101. This laser scan point cloud is obtained as a data set that collects data for each scan point, starting from the origin of the optical system of the laser scanner 101, including the distance and direction from that point, and the emission time of the corresponding scan light.

[0039] While the heavy machinery 100 is performing its work, the laser scanner 101 performs a laser scan to obtain a laser scan point cloud of the object that changes due to the work (for example, the terrain being excavated). However, if the position and orientation of the laser scanner 101 change due to the movement of the heavy machinery 100 or the rotation of the rotating part 110, the origin of the laser scan (the starting point mentioned above) will move, and the orientation of the coordinate axes of the laser scanner 101 will change, making it impossible to describe the laser scan point cloud on a single coordinate system. Therefore, a coordinate transformation, as described later, is performed so that each point in the laser scan point cloud obtained by the laser scanner 101 can be handled on the same coordinate system.

[0040] The unnecessary data removal unit 302 removes laser scan point clouds of parts of the heavy machinery 100 that are unnecessary for creating 3D data of the work object, such as the arm 151 and the bucket 152. The laser scan point cloud obtained by the laser scanner 101 includes those of the arm 151 and the bucket 152 of the heavy machinery 100. These laser scan point clouds are unnecessary for generating 3D data of the work object, so the corresponding laser scan point clouds are removed.

[0041] For example, the laser scan point clouds of the arm 151 and bucket 152 move relative to the laser scan point cloud of the background terrain when the heavy machine 100 moves or the rotating part 110 rotates. For example, when the heavy machine 100 moves, the background laser scan point cloud moves in the opposite direction to the direction of movement, but the laser scan point clouds of the arm 151 and bucket 152 move in the same direction as the heavy machine 100 relative to the background point cloud. Point clouds exhibiting this behavior are identified as the point clouds of the arm 151 and bucket 152 and deleted.

[0042] The positioning data acquisition unit 303 acquires data on the position of the reflecting prism 102 measured by the total station 200. The total station 200 is equipped with a clock and also acquires the time when the position of the reflecting prism 102 is measured. The data on the position of the reflecting prism 102 is output from the total station 200 in association with the measurement time, and this is acquired by the positioning data acquisition unit 303.

[0043] The positioning data from Total Station 200 is described using the coordinate system from which the position and orientation of Total Station 200 were acquired. For example, if the position and orientation of Total Station 200 are acquired in an absolute coordinate system, the resulting positioning data will be described using the absolute coordinate system. Alternatively, if the position and orientation of Total Station 200 are acquired in a local coordinate system, the resulting positioning data will be described using that local coordinate system.

[0044] The relative positions of the laser scanner 101, the reflective prism 102, and the IMU 112 are known. Furthermore, the relationship between the attitudes of the laser scanner 101 and the IMU 112 is also known. Therefore, the position of the laser scanner 101 can be calculated using the positioning data of the reflective prism 102 measured by the total station 200 and the attitude data measured by the IMU 112. In other words, the position of the laser scanner 101 can be measured using the total station 200. For example, if the position and attitude of the total station 200 are obtained in an absolute coordinate system, the position of the laser scanner 101 in the absolute coordinate system can be measured using the total station 200.

[0045] The operating status acquisition unit 304 acquires information regarding whether the heavy machinery 100 is moving and whether the rotating part 110 is rotating. Here, whether the heavy machinery 100 is moving is determined by a signal from the movement detection device 111, and whether the rotating part 110 is rotating is determined by a signal from the rotation detection unit 112.

[0046] The posture data acquisition unit 306 acquires posture data measured by the IMU 112. The relationship between the position and posture of the IMU 112 and the laser scanner 101 is known. Therefore, the posture of the laser scanner 101 can be measured by the IMU 112.

[0047] The coordinate transformation unit 207 transforms the laser scan point cloud acquired by the laser scanner 101 into a specific coordinate system.

[0048] The data of the laser scan point measured by the laser scanner 101 consists of the distance and direction from the laser scanner 101. This data is described on the local coordinate system specific to the laser scanner 101.

[0049] When a laser scan is performed by the laser scanner 101 while the heavy machinery 100 is moving or the rotating part 110 is rotating, the origin of the laser scanner 101's unique local coordinate system also moves during the laser scan. Furthermore, this local coordinate system may rotate (change its orientation). In this case, the resulting laser scan point cloud cannot be described on a single coordinate system. In extreme cases, a unique coordinate system may be required for each of the numerous scan points (of course, this is not the case if there is no movement or if the movement is slow).

[0050] Therefore, the laser scan point cloud obtained by the laser scanner 101 is transformed into a specific coordinate system so that the point cloud data can be described on a single coordinate system. Here, we will explain the case where the laser scan point cloud acquired by the laser scanner 101 is transformed into an absolute coordinate system.

[0051] First, as a prerequisite, the total station 200 acquires its position and orientation in the absolute coordinate system beforehand. While the heavy machinery 100 is operating, the total station 200 continuously measures the position of the reflecting prism 102. The positioning data obtained at this time is obtained by associating the position in the absolute coordinate system with the positioning time data.

[0052] On the other hand, the IMU 112 measures the attitude of the laser scanner 101 in absolute coordinate system. The time of this attitude measurement is also acquired.

[0053] In this way, the position and orientation of the laser scanner 101 in the absolute coordinate system at a certain time t are obtained.

[0054] Therefore, each scan point obtained by the laser scanner 101 is transformed into an absolute coordinate system. For example, consider a scan point obtained by the laser scanner 101 at a certain time t. Here, the position of the laser scanner 101 in the absolute coordinate system at time t is measured by the total station 200, and its attitude in the absolute coordinate system is measured by the IMU 112. Thus, the position of the above scan point in the absolute coordinate system can be determined. By performing this operation for each scan point, each scan point obtained by the laser scanner 101 can be transformed into an absolute coordinate system.

[0055] Based on the above principle, it becomes possible to transform the coordinates of a laser scan point cloud, described on the local coordinate system specific to the moving and rotating laser scanner 101, onto an absolute coordinate system.

[0056] Generally, coordinate transformation from a first coordinate system to a second coordinate system is performed by translation and rotation. By providing the position of the laser scanner 101 in the absolute coordinate system, information regarding the translation is obtained, and by providing the orientation of the laser scanner 101 in the absolute coordinate system at that position, information regarding the rotation is obtained. In this way, a coordinate transformation is performed from the local coordinate system specific to the laser scanner 101 to the absolute coordinate system for the scan point measured at time t. By performing this coordinate transformation for each scan point, the laser scan point cloud obtained by the laser scanner 101 performing a laser scan while moving can be transformed into the absolute coordinate system. In this way, point cloud data is obtained that describes the laser scan point cloud obtained by the laser scanner 101 on the absolute coordinate system.

[0057] When transforming the laser scan point cloud obtained by the laser scanner 101 to a specific local coordinate system, the position and orientation of the laser scanner 101 are those of the local coordinate system being used. The IMU 112 also measures orientation data in that local coordinate system.

[0058] In this case, the position and attitude of the total station 200 are acquired in advance on a local coordinate system. Furthermore, the calibration of the IMU 112 and the calculation of its attitude in the local coordinate system are performed using the positioning data of the reflecting prism 102 from the total station 200, and the attitude data is obtained in the local coordinate system. Then, the position and attitude of the laser scanner 101 in the local coordinate system are acquired, and the above coordinate transformation is performed for each point.

[0059] In this case, the attitude data obtained by the IMU 112 and the positioning data of the reflecting prism 102 obtained by the total station 200 are sent to the arithmetic unit 300, where the arithmetic unit 300 performs calibration of the IMU 112 and calculates the attitude in the local coordinate system. This calculation is performed, for example, in the attitude data acquisition unit 306. This calculation may also be performed in the IMU 112.

[0060] Ideally, each laser scan point should be associated with time information, but it is also acceptable to capture time over a certain time range. For example, the measurement time for scan points P1, P2, P3, P4, and P5 could be time t1, and the measurement time for scan points P6, P7, P8, P9, and P10 could be time t2.

[0061] Since laser scanning is performed dot by dot, when considering multiple points aligned on the time axis, the set of these points is obtained over a certain time span. Therefore, when considering point cloud data within a certain range, the time at which the last point was measured is defined as the acquisition time of that point cloud data. In other words, the last time within that time span is defined as the acquisition time of the point cloud data. It is also possible to use the first time or an intermediate time within that time span as the acquisition time of the point cloud data.

[0062] The coordinate-transformed point cloud data acquisition unit 308 acquires point cloud data that has been transformed to the specific coordinate system described above. This point cloud data includes point cloud data of the terrain on which the heavy machinery 100 is working.

[0063] The change detection unit 310 detects changes in 3D data by comparing 3D data moving forward and backward on the time axis and detecting the difference. The 3D data is point cloud data that has been coordinate-transformed to the specific coordinate system described above, or a 3D model created based on the point cloud data, as described later.

[0064] The 3D model creation unit 310 creates a 3D model based on the point cloud data acquired by the coordinate-transformed point cloud data acquisition unit 308. The 3D model is an outline model or TIN model created based on the point cloud data. For example, 3D models used in CAD are one such example.

[0065] The data storage unit 311 stores various data and operating programs used by the calculation unit 300, as well as various data obtained by the calculation unit 300 (for example, 3D data such as point cloud data).

[0066] The work content identification unit 312 identifies the content of the work performed by the heavy machinery 100 based on the point cloud data acquired by the coordinate-transformed point cloud data acquisition unit 308 or a 3D model based on said point cloud data. At this time, 3D data (point cloud data or 3D model) of monitored objects such as people, as described later, is excluded because it does not represent the content of the work.

[0067] Specifically, 3D data based on images taken at a specific stage before a particular operation is compared with 3D data based on images taken at a specific stage after a particular operation, and the difference data is detected. This process is performed in the change detection unit 309. Next, from the above difference data, 3D data of objects that move on the ground and move relative to the heavy machinery 100 are removed as 3D data of monitored targets such as people. This process is performed in the monitored target identification unit 312, which will be described later. In this way, the changes in 3D data caused by the operation of the heavy machinery 100 are detected, and the content of the operation (such as cutting the ground) is identified.

[0068] The following is a specific example. The terrain changes as heavy machinery 100 performs civil engineering work. Figure 2(A) shows the state of the civil engineering construction site at time t1, and Figure 2(B) shows the state of the civil engineering construction site at time t2. Here, t2 is a later time than t1.

[0069] Figure 2(A) (time t1) shows the state in which topographic elevations 401 and 402 exist. Figure 2(B) (time t2) shows the state after time t1 in Figure 2(A), where topographic elevation 401 has been removed by heavy machinery 100 to make the surface flat, and topographic elevation 402 remains.

[0070] Focusing on the terrain, as shown in Figure 3(B), the work of the heavy machinery 100 causes the topographic elevation 401 shown in Figure 3(A) to disappear.

[0071] Now, focusing on the 3D terrain data in the state shown in Figure 2(A) or Figure 3(A) and the 3D terrain data in the state shown in Figure 2(B) or Figure 3(B), there is a difference in the 3D data corresponding to the terrain elevation 401 between the two.

[0072] In this context, 3D data refers to point cloud data or a 3D model based on said point cloud data.

[0073] In the example above, by calculating the difference between the 3D terrain data for the situation in Figure 3(B) and the 3D terrain data for the situation in Figure 3(A), the 3D data of the terrain elevation 401 that was to be removed can be extracted.

[0074] Furthermore, the laser scan performed by the laser scanner 101 is not performed at a single point on the time axis, but rather over a certain time span. Therefore, the time associated with the obtained 3D data has a certain time span and can be determined, for example, by its intermediate or representative value.

[0075] For example, 3D data is updated every 10 or 30 seconds. During this update, newly added 3D data and lost 3D data (for example, the 3D data for terrain elevation 401 mentioned above) can be extracted and saved separately. The update interval can be set as appropriate depending on the capabilities of the hardware being used, the required resolution on the time axis, etc.

[0076] For example, let's assume that point cloud data is updated every 30 seconds. In this case, by focusing on the point cloud data at a certain time, we can extract the point cloud data that is the difference between that data and the point cloud data from 30 seconds prior, thereby obtaining newly added and deleted point cloud data. For example, one example of deleted point cloud data is the point cloud data for "topographic elevation 401," which is ultimately removed, as shown in Figure 4.

[0077] For example, 3D data is generated for each of the following time points: T1, T2, T3, etc. In this case, the 3D data for each time point is stored in association with that time point. This data is stored in the data storage unit 311.

[0078] By tracking the time-series progression of 3D data at each of the above time points, changes in the 3D data can be understood. For example, changes in terrain caused by civil engineering work by heavy machinery 100 can be understood as changes in 3D data. For example, changes in terrain caused by civil engineering work by heavy machinery 100 can be displayed as an image on a screen as a time-series change of the 3D model. In addition, excavated and piled-up soil can be understood as 3D data.

[0079] The object of change is not limited to terrain; it may also be buildings or structures. For example, by applying this embodiment when demolishing buildings or structures with heavy machinery, the changes can be obtained as 3D data.

[0080] The monitoring target identification unit 312 identifies objects that move on the ground relative to the heavy machinery 100, particularly objects that approach the heavy machinery 100 on the ground, as monitoring targets. By performing this process periodically, it is possible to avoid proximity or contact between the heavy machinery 100 and people, and proximity or contact between the heavy machinery 100 and other heavy machinery.

[0081] Furthermore, since the 3D data subject to the above monitoring is not the 3D data of the object of the heavy machinery 100's work, identifying it allows for the extraction of 3D data of the object of the work (e.g., terrain) as shown in Figure 3, enabling efficient identification of the work content. For example, it becomes possible to distinguish between people and the object of the work on the 3D data.

[0082] For example, let's consider a scenario where a person approaches heavy machinery 100. Suppose the 3D data is updated every 5 seconds. In this case, by monitoring the changes in the 3D data every 5 seconds, the presence approaching heavy machinery 100 can be detected as a target for monitoring.

[0083] Here, changes in acquired 3D data are monitored, and 3D data that moves relative to the background 3D data (3D terrain data) and also relative to the heavy machinery 100 is detected as a target for monitoring. In other words, 3D data moving on the ground is detected. Furthermore, by detecting whether the target of monitoring is a person using image recognition processing, it is possible to understand the situation when a person approaches the heavy machinery 100. Note that if notification is to be provided as described later, the processing related to identifying the target of monitoring must be performed in real time or with as little delay as possible.

[0084] The notification unit 314 provides notification regarding the monitored object identified by the monitored object identification unit 313. For example, it provides notification when the distance between the heavy machinery 100 and a monitored object such as a person falls below a predetermined distance. Notification is provided in the form of sound such as an alarm sound to the operator of the heavy machinery 100 and the surrounding area, notification display using a display placed in the driver's seat of the heavy machinery 100, or output of a notification signal to an external device such as a smartphone.

[0085] (Example of processing procedure: Part 1) Figure 5 shows an example of the processing procedure. The program that executes the processing in Figure 5 is stored in the data storage unit 311 or a suitable storage medium, read from there, and executed by the CPU of the computer that constitutes the arithmetic unit 300. It is also possible to store the program on a server connected to the Internet and download it from there. The same applies to the processing shown in Figure 6.

[0086] The process shown in Figure 5 may be performed simultaneously with the operation of the heavy machinery 100, or it may be performed as post-processing after the completion of work by the heavy machinery 100.

[0087] First, the laser scan point cloud data obtained by the laser scanner 101 is acquired (step S101). Next, unnecessary point cloud data is removed from the acquired laser scan point cloud data (step S102). Here, the unnecessary point cloud data is the scan data of parts that are not needed to create 3D data of the work object, such as the arm 151 and bucket 152 of the heavy machinery 100. This process is performed by the unnecessary data removal unit 302.

[0088] Next, data on the position of the reflecting prism 102, as measured by the total station 200, is acquired (step S103). Then, data on the attitude, as measured by the IMU 112, is acquired (step S104).

[0089] Next, the laser scan point cloud obtained in step S101 is transformed to a specific coordinate system (step S105). This process is performed by the coordinate transformation unit 307. Then, the transformed point cloud data is acquired as "point cloud data in a specific coordinate system" (step S106).

[0090] Next, two point cloud data points that are in a different order on the time axis are compared, the difference is taken, and the point cloud data that has changed over time is detected (step S107). This process is performed by the change detection unit 309.

[0091] Next, point cloud data relating to objects moving on the ground is removed from the point cloud data in which changes have been detected, and the point cloud data that has changed due to the work of the heavy machinery 100 is identified as point cloud data relating to the work content (step S108). Here, the identification of point cloud data relating to objects moving on the ground in the point cloud data in which changes have been detected is performed by the monitoring target identification unit 313. In addition, the process of removing point cloud data relating to objects moving on the ground from the point cloud data in which changes have been detected and identifying point cloud data relating to the work content is performed by the work content identification unit 312.

[0092] (Example of processing procedure: Part 2) Figure 6 is a flowchart showing an example of the procedure for preventing interference between the heavy machinery 100 and people, etc. This procedure needs to be performed with as little delay as possible from the acquisition of the laser scan point cloud. First, the procedures in steps S101 to S106 in Figure 5 are performed to obtain the laser scan point cloud data obtained by the laser scanner 101 (step S201).

[0093] Next, objects that move relative to the heavy machinery 101 and the background terrain are identified as monitoring targets (step S202). This process is performed by the monitoring target identification unit 313. Next, the distance between the heavy machinery 100 and the monitoring target is calculated, and it is determined whether or not this distance is less than or equal to a predetermined distance (step S203).

[0094] If the above distance is less than or equal to a predetermined distance, a notification process is performed (step S204); otherwise, the process from step S201 onwards is repeated.

[0095] (Superiority) In this embodiment, measurement data is received while tracking the position of the reflective prism 102 of a heavy machine 100 equipped with a laser scanner 101 and a reflective prism 102 using a total station 200 whose position and orientation in a specific coordinate system are known. A laser scan point cloud obtained by the laser scanner 101 is acquired. Based on the measurement data and the orientation of the laser scanner 200, the laser scan point cloud is transformed to the specific coordinate system. Changes in the point cloud data obtained by the coordinate transformation on the time axis are detected, and the content of the work performed by the heavy machine is identified based on the changes in the point cloud data on the time axis.

[0096] According to this embodiment, 3D data of the work object can be obtained simultaneously with the operation of the heavy machinery. Therefore, the work involved in digitizing the results of the work performed by the heavy machinery is made more efficient. In addition, it is possible to detect the risk of interference between the heavy machinery and workers, avoid dangerous situations that could lead to accidents, and even prevent accidents.

[0097] 2. Second Embodiment The position of the laser scanner 101 may also be obtained from the measurement values ​​of the GNSS device 111. In this case, the positioning data of the GNSS device 111 is calibrated based on the positioning data of the reflecting prism 102 from the total station 200, and the calibrated measurement values ​​are used. The positioning data of the GNSS device 111 alone contains errors, but by performing the above calibration using the total station 200, the accuracy of the position measurement by the GNSS device 111 can be improved.

[0098] Furthermore, by performing positioning using a GNSS device 111 that utilizes relative positioning methods such as RTK, it is possible to eliminate the need for positioning of the reflecting prism 102 by the total station 200 (of course, both methods can be used in combination). In this case, the positioning data acquisition unit 303 acquires the position information measured by the GNSS device 111.

[0099] 3. Third Embodiment It is also possible to use a data processing server to execute the functions of the calculation unit 300. In this case, various measurement data from the heavy machinery 100 and the total station 200 are transmitted to the data processing server via an appropriate data communication line such as an internet connection, and the processing performed by the calculation unit 300 is executed there.

[0100] 4. Fourth Embodiment Another method involves selecting and acquiring a laser scan point cloud obtained while the laser scanner 101 is stationary (not moving) relative to the ground (absolute coordinate system), and then using this acquired laser scan point cloud to obtain point cloud data of the target of the heavy machinery 100's work.

[0101] The operating status acquisition unit 304 acquires data necessary to determine whether the heavy machinery 100 is moving or not, and whether the rotating part 110 is rotating or not. For example, suppose the heavy machinery 100 is not moving and the rotation of the rotating part 110 has stopped between times t1 and t2. In this case, the laser scan point cloud obtained by the laser scanner 101 between times t1 and t2 is used for calculations related to identifying the work content and identifying the target of monitoring.

[0102] For example, suppose we have obtained a first laser scan point cloud obtained by a laser scanner 101 stationary at a first position and in a first orientation, a second laser scan point cloud obtained by a laser scanner 101 stationary at a second position and in a second orientation, a third laser scan point cloud obtained by a laser scanner 101 stationary at a third position and in a third orientation, and so on.

[0103] Here, the position of the origin of the laser scan data obtained in each stationary state (the position of the optical origin of the laser scanner 101) is determined from the position of the reflecting prism 01, which is measured by the total station 200. Since the positional relationship between the laser scanner 101 and the reflecting prism 102 is known, the position of the laser scanner 101 can be determined based on the position of the reflecting prism 102 and the attitude data obtained from the IMU 112. Furthermore, the attitude of the laser scanner 101 can be determined based on the attitude data obtained from the IMU 112.

[0104] The specific coordinate system used here is the same coordinate system used to determine the position and orientation of the total station 200, as described in the first embodiment.

[0105] In this way, the position and orientation (direction) of the first laser scan point cloud, the position and orientation of the second laser scan point cloud, the position and orientation of the third laser scan point cloud, etc., in a specific coordinate system are obtained. Then, each laser scan point cloud can be transformed to a specific coordinate system, and multiple laser scan point clouds described on the same coordinate system are obtained.

[0106] Whether the heavy machinery 100 is moving or not, and whether the rotating part 110 is rotating or not, may be determined from the positioning data of the reflecting prism 102 measured by the total station 200. For example, suppose the total station measures the position at 0.5-second intervals. In this measurement of the position of the reflecting prism 102 performed at 0.5-second intervals, if there is no change in the measured position of the reflecting prism 102 at adjacent positioning times t1 and t2 on the time axis, it is determined that the heavy machinery 100 is not moving and the rotating part 110 is not rotating during that period (between times t1 and t2).

[0107] 5. Others This invention can also be applied to the installation and assembly of buildings and structures using heavy machinery, and to the demolition of buildings and structures using heavy machinery. [Explanation of Symbols]

[0108] 100...Heavy machinery, 101...Laser scanner, 102...Reflective prism, 103...Camera, 120...Base unit, 110...Rotating unit, 111...GNSS device, 112...IMU, 113...Travel detection device, 114...Rotation detection device, 300...Computation unit.

Claims

1. A measurement data receiving unit that receives measurement data while tracking the position of the reflective prism of a heavy machine equipped with a laser scanner and a reflective prism using a surveying device whose position and orientation on a specific coordinate system are known, A laser scan point cloud acquisition unit that acquires the laser scan point cloud obtained by the laser scanner, A coordinate transformation unit that transforms the laser scan point cloud into a specific coordinate system based on the measurement data and the orientation of the laser scanner, A change detection unit detects changes in the time axis of the point cloud data obtained by the coordinate transformation as difference data, From the aforementioned difference data, a work content identification unit detects the changes in the point cloud data caused by the operation of the heavy machinery by removing point cloud data of objects that move on the ground and move relative to the heavy machinery, thereby identifying the content of the operation of the heavy machinery. A computing device equipped with the following features.

2. The calculation device according to claim 1, further comprising a monitoring target detection unit that detects an object moving on the ground toward the heavy machine as a monitoring target based on the changes in the point cloud data on the time axis.

3. The calculation device according to claim 2, further comprising a notification unit that performs notification processing when the monitored object approaches the heavy machinery to a predetermined distance or less.

4. The laser scan point cloud includes a point cloud targeting a part of the heavy machinery. The calculation device according to claim 1, further comprising a removal unit for removing the point cloud from a part of the heavy machinery.

5. The system receives measurement data while tracking the position of the reflective prism of a heavy machine equipped with a laser scanner and a reflective prism using a surveying device whose position and orientation on a specific coordinate system are known. The laser scanner acquires the laser scan point cloud obtained by the aforementioned laser scanner, Based on the measurement data and the orientation of the laser scanner, the laser scan point cloud is transformed into the specific coordinate system. The changes in the point cloud data obtained by the aforementioned coordinate transformation on the time axis are detected as difference data. A calculation method for detecting changes in the point cloud data caused by the operation of the heavy machinery and identifying the content of the operation of the heavy machinery by removing point cloud data of objects that move on the ground and move relative to the heavy machinery from the difference data.

6. It is a program that is read and executed by a computer. Computer A measurement data receiving unit that receives measurement data while tracking the position of the reflective prism of a heavy machine equipped with a laser scanner and a reflective prism using a surveying device whose position and orientation on a specific coordinate system are known, A laser scan point cloud acquisition unit that acquires the laser scan point cloud obtained by the laser scanner, A coordinate transformation unit that transforms the laser scan point cloud into a specific coordinate system based on the measurement data and the orientation of the laser scanner, A change detection unit detects changes in the time axis of the point cloud data obtained by the coordinate transformation as difference data, From the aforementioned difference data, a work content identification unit detects the changes in the point cloud data caused by the operation of the heavy machinery by removing point cloud data of objects that move on the ground and move relative to the heavy machinery, thereby identifying the content of the operation of the heavy machinery. A program that is used to make something work.

7. A positioning data receiving unit that receives positioning data from a GNSS positioning device mounted on heavy machinery, A laser scan point cloud acquisition unit acquires a laser scan point cloud obtained by a laser scanner mounted on the aforementioned heavy machinery, A coordinate transformation unit that transforms the laser scan point cloud into an absolute coordinate system based on the positioning data and the attitude of the laser scanner, A change detection unit detects changes in the time axis of the point cloud data obtained by the coordinate transformation as difference data, From the aforementioned difference data, a work content identification unit detects the changes in the point cloud data caused by the operation of the heavy machinery by removing point cloud data of objects that move on the ground and move relative to the heavy machinery, thereby identifying the content of the operation of the heavy machinery. A computing device equipped with the following features.

Images