Methods for acquiring information on heavy machinery, devices and programs for handling information on heavy machinery

Abstract

This system allows for easy acquisition of information regarding the relative positions of a camera positioned on the rotating body of heavy machinery and an optical target. [Solution] A method for acquiring information on heavy machinery 100, comprising: a first step of measuring the position of the reflective prism 107 with a total station 300 and performing SfM using images captured by a camera 108 during the operation of rotating the slewing body 103; a second step of measuring the position of the reflective prism 107 with a total station 300 and performing SfM using images captured by a camera 108 during the operation of moving the traveling body 101 in a straight line without rotating the slewing body 103; and determining the relationship between the position of the reflective prism 107 on the slewing body 103 and the position of the camera 108 based on the position of the reflective prism 107 obtained in the first and second steps and the position and orientation of the camera 108 obtained by SfM.

Description

[Technical Field]

[0001] This invention relates to a technology for handling information about heavy machinery. [Background technology]

[0002] Techniques for measuring the position of movable parts of heavy machinery are known (see, for example, Patent Documents 1-4). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2017-181340 [Patent Document 2] U.S. Patent Publication No. 6711838 [Patent Document 3] Patent No. 7514459 [Patent Document 4] Japanese Patent Publication No. 2008-2842 [Overview of the project] [Problems that the invention aims to solve]

[0004] One technique for detecting the position of the cutting edge of heavy machinery (construction equipment), such as the bucket of a hydraulic shovel, using surveying equipment, involves placing a reflective prism and a camera on the rotating body of the hydraulic shovel. The position of the reflective prism is measured using a total station, while the camera's position and orientation are calculated using SFM (Surface Frequency Modeling) based on the image captured by the camera. By performing adjustment calculations, the absolute position and orientation of the rotation center of the rotating body can be determined, thereby calculating the position of the cutting edge.

[0005] This technology requires preparatory work to precisely determine the relationship between the camera position and the reflective prism position on the rotating body. This work is complicated, and there is a need for a technology that can be performed simply. Against this backdrop, the present invention aims to provide a technology that can easily acquire information on the positional relationship between a camera and an optical target placed on the rotating body of heavy machinery. [Means for solving the problem]

[0006] The present invention relates to a method for acquiring information about a heavy machine having a mobile body equipped with a means of transport and a rotating body that is rotatable on the mobile body and equipped with an optical target and a camera, comprising: a first step of measuring the position of the optical target with a surveying device and performing SfM using images captured by the camera during the process of rotating the rotating body; a second step of measuring the position of the optical target with the surveying device and performing SfM using images captured by the camera during the process of moving the mobile body in a straight line without rotating the rotating body; and a method for acquiring information about a heavy machine in which the relationship between the position of the optical target and the position of the camera on the rotating body can be determined based on the position of the optical target obtained by the surveying device and the position and orientation of the camera obtained by SfM in the first and second steps.

[0007] The present invention relates to a method for acquiring information about a heavy machine having a mobile body equipped with a means of transport and a rotating body rotatable on the mobile body and equipped with an optical target and a camera, comprising: a first step of measuring the position of the optical target with a surveying device and performing SfM using images captured by the camera during the process of rotating the rotating body; a second step of measuring the position of the optical target with the surveying device and performing SfM using images captured by the camera during the process of moving the mobile body in a straight line without rotating the rotating body; a third step of measuring the position of the optical target with the surveying device and performing SfM using images captured by the camera during the process of tilting the rotating body from the horizontal; and a third step of determining the relationship between the position of the optical target and the position of the camera on the rotating body based on the position of the optical target obtained by the surveying device and the position and orientation of the camera obtained by SfM in the first to third steps.

[0008] In the present invention, bundle adjustment calculations are performed in the first and second steps based on images captured by the camera, and the bundle adjustment calculations are performed with the constraint that the difference between the position of the optical target obtained by the surveying device in the first and second steps and the position of the optical target obtained by SfM is minimized.

[0009] In the present invention, the heavy machine is a hydraulic shovel, and in the third step, the slewing body is tilted from the horizontal direction by pushing the ground with the bucket provided on the hydraulic shovel.

[0010] The present invention relates to a device for handling information of heavy machinery having a mobile body equipped with a means of movement and a rotating body rotatable on the mobile body and equipped with an optical target and a camera, wherein the device includes a calculation unit that performs the following steps: a first step in the process of rotating the rotating body, in which a surveying device measures the position of the optical target and performs SfM using images captured by the camera; and a second step in the process of moving the mobile body in a straight line without rotating the rotating body, in which the surveying device measures the position of the optical target and performs SfM using images captured by the camera, and determines the relationship between the position of the optical target and the position of the camera on the rotating body based on the position of the optical target obtained by the surveying device and the position and orientation of the camera obtained by SfM obtained in the first and second steps.

[0011] The present invention is a program that causes a computer to process information about a heavy machine having a mobile body equipped with a means of movement and a rotating body that is rotatable on the mobile body and equipped with an optical target and a camera, wherein the program causes the computer to perform a first step in which, during the operation of rotating the rotating body, the position of the optical target is measured by a surveying device and SfM is performed using the image captured by the camera, and a second step in which, during the operation of moving the mobile body in a straight line without rotating the rotating body, the position of the optical target is measured by the surveying device and SfM is performed using the image captured by the camera, and the relationship between the position of the optical target and the position of the camera on the rotating body is determined based on the position of the optical target obtained by the surveying device and the position and orientation of the camera obtained by SfM obtained in the first and second steps. [Effects of the Invention]

[0012] According to the present invention, information regarding the positional relationship between a camera placed on the rotating body of heavy machinery and an optical target can be easily acquired. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram of an embodiment. [Figure 2] This is a block diagram of the arithmetic unit. [Figure 3] This is a flowchart showing an example of the processing procedure. [Figure 4] This is a flowchart showing an example of the processing procedure. [Figure 5] This is a conceptual diagram showing the relationship between the camera position and the prism position. [Figure 6] This is a conceptual diagram of an embodiment. [Figure 7] This is a flowchart showing an example of the processing procedure. [Modes for carrying out the invention]

[0014] 1. First Embodiment (Summary) In the configuration shown in Figure 1, the total station 300 continuously measures the position of the reflecting prism 107 in the absolute coordinate system (also called the global coordinate system). The absolute coordinate system is the coordinate system used in GNSS and maps. For example, a position in the absolute coordinate system is described by longitude, latitude, and elevation.

[0015] The heavy machinery 100 performs on-site calibration at the start of work. In this process, the rotating body 103 is rotated while the traveling body 101 is stationary, and the rotation center position of the rotating body 103 in the absolute coordinate system is determined from the movement trajectory of the reflective prism 107 during this rotation.

[0016] If the mobile unit 101 does not move from the time of on-site calibration, the rotation center position of the slewing unit 103 does not move, and the rotation center position of the slewing unit 103 is determined from the measurement position of the reflecting prism 107 by the total station 300. Based on the position of the reflecting prism 107, the rotation center position of the slewing unit 103, and the measurement values ​​of the tilt sensors 113 to 116 described later, the position and orientation of the cutting edge 106a of the bucket 106 are calculated.

[0017] As the mobile unit 101 moves, the rotation center position of the rotating unit 103 and the camera 108 also move. The movement path of the camera 108 at this time is calculated by SfM (Structure from Motion) based on the images captured by the camera 108. In addition, the position of the reflective prism 107 is tracked by a total station, and its position is continuously measured.

[0018] Here, the positional relationship between the reflective prism 107, the camera 108, and the rotation center of the rotating body 103 is known through the initial calibration described later. Therefore, based on the movement path of the reflective prism 107 during the movement of the mobile body 101, the movement path of the camera 108 obtained by SfM, and the results of the initial calibration described above, the movement path of the rotation center position of the rotating body 103 in the absolute coordinate system during the movement of the mobile body 101 can be determined.

[0019] As a result, even if the mobile body 101 moves from the time of on-site calibration, the determination of the rotation center position of the rotating body 103 in the absolute coordinate system continues. This continues even if the mobile body 101 moves intermittently. Therefore, even if the mobile body 101 moves, it is still possible to calculate the position and orientation of the cutting edge 106a based on the measurement position of the reflecting prism 107 by the total station 300.

[0020] (Heavy machinery) Figure 1 shows a heavy machine 100, which is a hydraulic shovel. The heavy machine 100 is equipped with a mobile body 101. The mobile body 101 is equipped with a continuous track 102, which is a means of movement. The mobile body 101 is equipped with a slewing body 103 on top of it that is capable of horizontal rotation.

[0021] The slewing body 103 is equipped with a boom 104. The boom 104 rotates at its connection point with the slewing body 103, and its tip moves in a circular orbit within a vertical plane. This movement causes the tip of the boom 104 to move vertically.

[0022] An arm 105 is attached to the tip of the boom 104. The arm 105 rotates at its connection point to the boom 104, and its tip moves along a circular orbit in a vertical plane. This movement causes the tip of the arm 105 to move toward or away from the slewing body 103.

[0023] A bucket 106 is attached to the tip of the arm 105. The bucket 106 rotates in a vertical plane around the point where it is attached to the arm 105 as its pivot point. This movement causes the tip of the bucket 106 (the cutting edge 106a) to move toward and away from the slewing body 103 on an arc centered on the aforementioned pivot point. The movement of each part is performed by hydraulics. The movement and driving method of each part are the same as those of a normal hydraulic shovel.

[0024] A reflective prism 107, which is an optical target, and a camera 108, which is a means of taking pictures, are mounted on the upper part of the rotating body 103. The reflective prism 107 is an all-around reflective prism that reflects light incident from a range of 360° horizontally and ±30° vertically, reversing its direction by 180°. The reflective prism 107 is mounted off-center from the rotation center of the rotating body 103. In other words, the reflective prism 107 is positioned to move along a circular orbit when the rotating body 103 rotates while the traveling body 101 is stationary. The installation position of the reflective prism 107 on the rotating body 103 does not need to be precisely determined, but its positional relationship to the rotation center of the rotating body 103, such as being on the front side or the rear side, may be predetermined.

[0025] Camera 108 repeatedly takes continuous shots at a specific frequency. The repetition frequency for shooting is set to 0.5 to 30 Hz. It is also possible to shoot video and use frame images. The position and orientation changes of camera 108 are calculated using Structure from Motion (SfM) based on the images captured by camera 108. It is also possible to use two or more cameras.

[0026] Camera 108 is positioned and positioned so that the ground is within its field of view. Camera 108 is used to calculate the rotation center position of the rotating body 103. Camera 108 can face any direction as long as it can photograph the ground, and its installation position is not limited (although it must be fixed to the rotating body 103). Alternatively, camera 108 may be pointed forward to photograph the work being done. In this case, image information recording the work being done can be obtained from camera 108.

[0027] The installation positions of the reflective prism 107 and camera 108 on the swivel body 103 do not need to be precisely positioned during installation. The precise positional relationship between the two can be determined during the adjustment calculations performed during the initial calibration. It is preferable to provide an approximate separation distance L between the two as a constraint during these adjustment calculations, as this reduces the computational burden and improves accuracy. Therefore, an approximate value of L may be determined or specified in advance when installing the reflective prism 107 and camera 108 on the swivel body 103 (for example, installing them approximately 1 meter apart).

[0028] A tilt sensor 113 is attached to the slewing body 103. A tilt sensor 114 is attached to the boom 104, a tilt sensor 115 is attached to the arm 105, and a tilt sensor 116 is attached to the bucket 106.

[0029] The heavy machinery 100 is equipped with a computing device 200. The computing device 200 is a computer and performs various calculations, which will be described later.

[0030] (T-Station) Total station 300 is set up with its position and orientation in the absolute coordinate system known. Total station 300 measures the distance to the reflecting prism 107 using the principle of laser ranging. By measuring the direction of the optical axis of the ranging laser beam at this time, the direction of the reflecting prism 107 as seen from total station 300 can be obtained.

[0031] Knowing the distance from the total station 300 to the reflecting prism 107 and the direction of the reflecting prism 107 as seen from the total station 300 allows us to determine the position of the reflecting prism 107 relative to the total station 300. On the other hand, the position and orientation of the total station 300 in absolute coordinates are known. Therefore, based on the measurement of the position of the reflecting prism 107 by the total station 300, the position of the reflecting prism 107 in absolute coordinates can be obtained.

[0032] The position of the reflecting prism 107 is measured repeatedly by the total station 300 at intervals of 10 Hz to 20 Hz. The total station 300 can also be equipped with a camera to photograph the object being measured.

[0033] The total station 300 has the function of searching for the reflecting prism 107, locking onto the sighted reflecting prism 107, and continuing to track and measure its position even if the reflecting prism 107 moves.

[0034] (Block diagram of the arithmetic unit) Figure 2 is a block diagram of the arithmetic unit 200. The arithmetic unit 200 is a computer and comprises a data reception unit 201, an SfM calculation unit 202, a reflective prism movement trajectory acquisition unit 203, a rotation center position calculation unit 204, an initial calibration processing calculation unit 205, a cutting edge position and orientation calculation unit 207, a storage unit 210, a total station operation control unit 211, and a communication device 212.

[0035] Functional units other than the memory unit 210 and the communication device 212 are configured in software, and the operation programs for executing these functions are realized by the CPU of the arithmetic unit 200 executing them. It is also possible to configure some or all of these functional units with dedicated hardware (electronic circuits).

[0036] The data reception unit 201 receives measurement data of the position of the reflecting prism 107 measured by the total station 300, and image data captured by the camera 108.

[0037] The SfM calculation unit 202 performs mutual orientation and absolute orientation based on the images captured by the camera 108, and calculates the position and orientation of the camera 108 at the time of capture for each captured image, as well as the positions of feature points extracted from the captured images.

[0038] Through the initial calibration described later, the rotation center positions of the reflective prism 107 and the slewing body 103, and the positional relationship of the camera 108 are obtained as known information (initial calibration values). In addition, the scale information necessary for absolute orientation is also obtained during the initial calibration. Furthermore, by measuring the position of the reflective prism 107 using the total station 300 during field calibration, the position of the reflective prism 107 in the absolute coordinate system (measured value) and the rotation center position of the slewing body 103 (calculated value) at the start of the operation are obtained.

[0039] Therefore, the position and orientation of camera 108 in the absolute coordinate system at the start of the operation (initial values ​​at the start of SfM) are given by using the position of the reflective prism 107, the rotation center position of the rotating body 103, and the initial calibration value. As a result, the position and orientation of camera 108 calculated by SfM are obtained in the absolute coordinate system. In addition, the positions of feature points obtained from the target object in the absolute coordinate system are also calculated.

[0040] The reflective prism movement trajectory acquisition unit 203 acquires the movement trajectory (change in position in absolute coordinate system) of the reflective prism 107 based on the position measurement data of the reflective prism 107 by the total station 300.

[0041] The rotation center position calculation unit 204 calculates the rotation center position of the rotating body 103 based on the trajectory of the moving reflective prism 107. When the rotating body 103 rotates while the traveling body 101 is stationary, the reflective prism 107 moves along a circular orbit. Therefore, if the position of the reflective prism 107 is continuously measured by the total station 300 at this time, the measured movement trajectory of the reflective prism 107 will be an arc, which is part of a circular orbit (if the rotating body 103 completes one rotation, it will be a circular orbit).

[0042] The axis passing through the center of curvature of this arc and perpendicular to the rotating body 103 becomes the axis of rotation of the rotating body 103. This axis of rotation can be calculated from the above arc. The center of rotation of the rotating body 103 lies on this axis of rotation. The position of the center of rotation of the rotating body 103 on this axis of rotation (position on the Z axis) is not limited to any predetermined position, but is set, for example, at the Z value position (height position) where the reflective prism 107 or camera 108 would be located, or inside the rotating body 103.

[0043] If the slewing body 103 is tilted, its axis of rotation will also be tilted. This tilt is measured by the tilt sensor 113. In the actual process, the tilt of the axis of rotation of the slewing body 103 is determined based on the tilt information from the tilt sensor 113, and the position of the center of rotation of the slewing body 103 on this axis of rotation is determined. Alternatively, the tilt of the axis of rotation of the slewing body 103 may be calculated from the tilt of the arc from the horizontal plane.

[0044] The initial calibration processing unit 205 performs calculations related to the initial calibration that is carried out after the reflective prism 107 and camera 108 are attached to the swivel body 103. In this process, the relationship between the position of the reflective prism 107, the rotation center position of the swivel body 103, the position of the camera 108, and the orientation of the swivel body is obtained as the initial calibration value. Details of the initial calibration will be described later.

[0045] The cutting edge position and orientation calculation unit 207 calculates the position and orientation of the cutting edge 106a of the bucket 106 based on the measurement position of the reflecting prism 107 by the total station 300, the rotation center position of the rotating body 103, and the measured values ​​of the tilt sensors 113, 114, 115, and 116.

[0046] Based on prior calibration processing and design data of the heavy machinery 100, the relationship between the position and orientation of the cutting edge 106a, the position of the reflecting prism 107, the rotation center position of the rotating body 103, and the measured values ​​of the tilt sensors 113, 114, 115, and 116 has been obtained.

[0047] Here, with the rotation center position of the rotating body 103 in the absolute coordinate system known, the position of the reflecting prism 107 in the absolute coordinate system is measured by the total station 300. From this information and the measurements from the tilt sensors 113 to 116, the position and orientation of the cutting edge 106a in the absolute coordinate system are calculated. This process is performed in the cutting edge position and orientation calculation unit 207.

[0048] The memory unit 210 stores data and operation programs necessary for the operation of the heavy machinery 100. The total station operation control unit 211 controls the start and stop of the measurement of the reflecting prism 107 by the total station 300. The communication device 212 communicates with the total station 300 and other equipment. Communication is performed using known wireless communication standards such as wireless LAN.

[0049] (Initial calibration) The details of the initial calibration process are described below. In the initial calibration, the heavy machine 100 is made to perform a predetermined operation, and based on the measurements obtained at that time, a predetermined calculation is performed to obtain the position of the reflective prism 107, the position of the rotation center of the rotating body 103, and the relationship between the position and orientation of the camera 108. The initial calibration provides initial values ​​for SfM based on the image captured by the camera 108, enabling the acquisition of the position and orientation of the camera 108 in an absolute coordinate system using SfM, and the acquisition of three-dimensional position information of the object being photographed in an absolute coordinate system.

[0050] Figure 3 is a flowchart showing the operation procedure (operation procedure) of the heavy machinery 100 during initial calibration. Figure 4 is a flowchart showing the calculation processing procedure. The program that executes the processing in Figure 4 is stored in the storage unit 210 or a suitable storage medium within the arithmetic unit 200 and executed by the CPU of the arithmetic unit 200. Figure 2 shows the initial calibration processing calculation unit 205 as a functional unit that performs the processing in Figure 4.

[0051] First, the Total Station 300 is set up at its mechanical point. Mechanical point setup is the process of setting up the Total Station 300 with its position and orientation in the absolute coordinate system already known.

[0052] For example, a total station 300 is set up, and the positions of multiple reference points whose positions in the absolute coordinate system are known are measured from its position. Then, the position and orientation of the total station 300 in the absolute coordinate system are determined by a resection method using the coordinate values ​​of the measured multiple reference points. This allows for the mechanical positioning of the total station 300.

[0053] After the total station 300 is set up, the total station 300 begins measuring the position of the reflecting prism 107. At the same time, the camera 108 begins taking images. The interval (repetition frequency) of the total station 300 measuring the position of the reflecting prism 107 and the interval (repetition frequency) of the camera 108 taking images may be the same or different. Initial calibration is then started. During the initial calibration, if possible, the heavy machinery 100 should be on a horizontal and flat surface.

[0054] First, the rotating body 103 is rotated while the traveling body 101 is stationary (Figure 3: Step S101). The rotation angle is within the range of 90° to 360°. During the above rotation, the position of the reflecting prism 107 is repeatedly measured by the total station 300 and repeatedly photographed by the camera 108.

[0055] Once the rotation of the rotating body 103 is complete, a three-dimensional reconstruction process is performed based on the data captured by the camera 108 (Figure 4: Step S201). In this process, mutual localization is performed to determine the position and orientation of the camera 108 at the time of stereo image capture, and the three-dimensional relative relationship of feature points in the stereo image. The camera 108 moves in accordance with the rotation of the rotating body 103. During this movement, multiple images are taken with different viewpoints, and multiple stereo images are obtained. Mutual localization is performed on these multiple stereo images as described above.

[0056] Next, the rotation center positions of the reflecting prism 107 and the rotating body 103, and the absolute positional relationship of the camera 108 are obtained using the least squares method with the prism position information (Figure 4: Step S202). In this process, the bundle adjustment calculation shown in Equation 1, described later, is performed under the constraint conditions shown in Equation 2, described later.

[0057] Here, the rotation center position of the slewing body 103 is first calculated based on the movement trajectory of the reflective prism 107 in step S101. That is, in step S101, the reflective prism 107 moves along a circular orbit. By determining the center of this circular orbit (the center of curvature of the arc), the rotation center position of the slewing body 103 is calculated.

[0058] The position of the reflecting prism 107 is determined by taking into account the difference (time series) between the measurement time of the reflecting prism 107 and the shooting time of the camera 108. This is the same for steps S204 and S206.

[0059] Here, we assume that the position of the i-th reflective prism 107, as measured by the total station 300, is slightly different from its position at the time of the corresponding image captured by the camera 108. This discrepancy is due to the asynchronous nature of the time information (clocks) used by the total station 300 and the camera 108.

[0060] Figure 5 shows the relationship between the position of the reflecting prism 107 and the position of the camera 108 (position during shooting). The solid circle (p i , T pi)(i=1,2,3··) represents the position of the reflecting prism 107 and the time of measurement, as measured by the total station 300 (surveying device).

[0061] The dashed circle (p) indicates the position of the reflecting prism 107 and the time of its measurement. ti , T pti ) represents the position and time of the reflecting prism 107 when considering the time series (taking into account the time synchronization discrepancy). This is the position and time of the reflecting prism 107 that would be measured if the time were synchronized. In other words, the solid circle (p i , T pi ) represents the actual position and time of the reflected prism 107, indicated by the dashed circle (p ti , T pti ) represents the position and time of the reflecting prism 107, which would be expected to be measured if synchronization were achieved.

[0062] This discrepancy is expected to be smaller than the difference between the i-th and i+1th positions of the reflecting prism 107 as measured by the total station 300. Since the difference between the i-th and i+1th positions of the reflecting prism 107 is minute, this discrepancy is expected to correspond to the difference between the positioning time of the reflecting prism 107 and the shooting time of the camera 108.

[0063] Specifically, we assume that the reflective prism 107 at the time of shooting is located at a position shifted by the above-mentioned amount from the i-th position of the reflective prism 107 measured by the total station 300 to the (i+1)-th position. This amount of shift is defined as a correction term proportional to the difference between the positioning time of the reflective prism 107 and the shooting time of the camera 108. In other words, we assume that the position of the prism 107 at the corresponding time of shooting is located at a position shifted by this correction term from the i-th prism position measured by the total station 300 to the (i+1)-th prism position.

[0064] In the bundle adjustment calculation, based on the collinearity condition that the light bundle (bundle) connecting the three points of the feature points obtained from the photographed image of the photographing object, the points on the photographed image, and the projection center must be on the same straight line, an observation equation of Equation (1) below is established for each light bundle of each image, and the coordinates (Xj, Yj, Zj) of the feature points and the parameters of the position and orientation of the camera 108 (X oi , Y oi , Z oi , a<​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​) and (a 11i ~a 33i ) is an unknown and can be determined by adjustment calculations using Mathematics 1. Note that (X oi ,Y oi ,Z oi ) and (a 11i ~a 33i The initial values ​​of (Xj,Yj,Zj) are obtained from the initial value adjustment described later. Furthermore, since the rough relationship between the local coordinate system and the absolute coordinate system is determined through initial value adjustment, the initial values ​​of (Xj,Yj,Zj) are also relatively close to the true values ​​in the absolute coordinate system, although they contain some error. Due to these factors, the convergence and accuracy of the adjustment calculation are improved.

[0069]

number

[0070] Here, -p' ti This is the position of the reflecting prism 107 in the absolute coordinate system as measured by the total station 300 (X pi ,Y pi ,Z pi ), is a function of the time of shooting by camera 101 and the positioning time of the reflecting prism 107, and includes a correction term proportional to the difference between the two times. This correction term determines the actual measured position of the reflecting prism 107 (X pi ,Y pi ,Z pi The position of the reflecting prism 107 synchronized with SfM is determined based on (-p'). ti -L') represents the camera position in the local coordinate system, taking into account the time series of synchronization errors calculated from the position measurements of the reflecting prism 107 by the total station 300.

[0071] Here, the local coordinate system is a coordinate system that describes the positional relationship between the position and orientation of camera 108 and the feature points extracted from the image captured by camera 108. The above constraint equation is set up on the local coordinate system, and the camera position (-p') is calculated considering the time series of synchronization errors calculated from the position measurements of the reflecting prism 107 by the total station 300. ti-L') and the camera position t', which is the final unknown variable. i This formula calculates the difference at each camera position and then sums them up.

[0072] Equation 2 is the position of camera 108 (-p') calculated from the position of the reflecting prism 107 measured by total station 300. ti -L') and the predicted position t' of camera 108 obtained by SfM. i This is a mathematical formula for quantitatively comparing and .

[0073] In the adjustment calculation in step S202, the condition for minimizing the difference shown by equation 2 is searched for. This allows us to find an approximate similarity relationship with respect to the unknown L', and to determine the approximate positional relationship between the reflecting prism 107 and the camera 108.

[0074] In the adjustment calculation using Mathematics 1 and 2, feature points (Xj, Yj, Zj) and external reference elements (X oi ,Y oi ,Z oi ,a 11i ~a 33i (rotation matrix indicating orientation) and (L X ,L Y ,L Z The residuals of equations 1 and 2 are calculated using (Xj,Yj,Zj),(X) as parameters. In this process, the residuals converge using the least squares method such that (Xj,Yj,Zj),(X oi ,Y oi ,Z oi ,a 11i ~a 33i ),(L X ,L Y ,L Z We search for combinations of ). Here, (L X ,L Y ,L Z ) represents the offset amount in the X, Y, and Z directions between the position of camera 108 and the position of reflective prism 107.

[0075] Specifically, the parameters (Xj, Yj, Zj), (X) are adjusted so that the residuals shown in equations 1 and 2 are small. oi ,Y oi ,Zoi ,a 11i ~a 33i ),(L X ,L Y ,L Z The process of adding a correction factor to (Xj, Yj, Zj) and simultaneously calculating equations 1 and 2 is repeated. Then, the unknown parameters (Xj, Yj, Zj) and (Xj) satisfy the convergence condition for equations 1 and 2. oi ,Y oi ,Z oi ,a 11i ~a 33i ),(L X ,L Y ,L Z We find the combination of ). The convergence conditions are that the residuals are sufficiently small and the change in the residuals from the previous calculation is sufficiently small (the change in the calculation result has converged).

[0076] Furthermore, if the distance L between the camera 108 and the reflecting prism 107 is known, then L 2 =L X 2 +L Y 2 +L Z 2 Therefore, this also becomes a constraint condition for the adjustment calculation described above.

[0077] The following describes the initial value setting process. Initial value setting involves determining the initial values ​​for the correspondence between the position and time of camera 108, described in the local coordinate system, and the position and time of reflective prism 107, described in the absolute coordinate system. These initial values ​​are then used for the aforementioned adjustment calculations. This initial value setting process can also be described as determining the approximate relationship between the local and absolute coordinate systems used in the mutual orientation described above.

[0078] An example of the initial value adjustment process will be described below. This technology is described in Japanese Patent Application No. 2023-058216. The initial value adjustment is performed by the following three-step process. In the first step of the process, attention is paid to the start of movement of the position of the camera 108 and the start of movement of the position of the reflection prism 107, and the correspondence relationship between the two positions is specified. In this process, the corresponding points (positions) of the two are specified by paying attention to the starting point of movement. Conversely, the corresponding points can also be specified by paying attention to the end point of movement (the point where it stops).

[0079] In this process, regarding the measured position of the reflection prism 107 by the total station 300 (position in the absolute coordinate system) and the calculated position of the camera 108 by SfM based on the captured image of the camera 108 (position in the local coordinate system), the correspondence relationship at the timing when the movement starts and / or ends is obtained.

[0080] After the first step of the process, the second step of the process is performed. In the second step of the process, a process that pays attention to the change in position is performed. In this case, the angles formed by three consecutive points are calculated for all points (all camera positions and reflection prism positions). Then, the angle formed on the camera side and the angle formed on the reflection prism side are compared, and the combination that minimizes the sum of the absolute values of the differences is searched for.

[0081] For example, the above-mentioned angle formed by the position of the camera 101 is θ c1 、θ c2 、θ c3 、θ c4 ···θ cn and the above-mentioned angle formed by the position of the reflection prism 102 is θ p1 、θ p2 、θ p3 、θ p4 ···θ pn Suppose it is. In this case, while changing the combination, θ c1 、θ c2 、θ c3 、θ c4 ···θ cn and θ p1 、θ p2 、θ p3 、θ p4 ···θpn We take the difference between each angle and find the sum of their absolute values. The combination of positions that minimizes this sum is adopted as the corresponding combination.

[0082] For example, θ p1 -θ c1 , θ p2 -θ c2 , θ p3 -θ c3 The sum of the absolute values ​​of ..., θ p2 -θ c1 , θ p3 -θ c2 , θ p4 -θ c3 The sum of the absolute values ​​of ..., θ p3 -θ c1 , θ p4 -θ c2 , θ p5 -θ c3 Calculate the sum of the absolute values ​​of all the terms and find the combination that minimizes that sum.

[0083] This method compares the change in the position of the reflecting prism 107 in the absolute coordinate system with the change in the position of the camera 108 in the local coordinate system, and identifies the correspondence between the two. Once this correspondence is determined, initial value adjustments can be performed with even greater precision than in the first stage.

[0084] In the second stage of processing, it is desirable that the calculation interval for the position of camera 108 and the measurement interval for the position of the reflecting prism 107 be as close as possible (ideally identical). After the second stage of processing, the third stage of processing is performed. In the third stage of processing, the correspondence between the camera position and the reflecting prism position is determined by the least squares method. In this process, a transformation matrix from the local coordinate system to the absolute coordinate system is obtained. Using this transformation matrix, the elements in the local coordinate system (feature points, position and orientation of camera 108) are transformed into the absolute coordinate system. The transformation matrix obtained here and the results of the transformation using this transformation matrix are used as initial values ​​for the adjustment calculation described above.

[0085] In step S202, the initial value adjustment stage described above is before the adjustment calculation, and the positional relationship between the reflecting prism 107 and the camera 108 is unknown. Therefore, in step S202, the coordinate transformation described in the third stage is performed on a horizontal plane that includes the position of the reflecting prism 107, with the rotation center of the rotating body 103 as the reference.

[0086] Through the three-stage alignment process described above, initial values ​​are obtained for the relationship between the local coordinate system, which describes the relationship between the camera position and feature points of each captured image obtained by mutual orientation, and the absolute coordinate system. The first and third stages described above are also performed in steps S204 and S206.

[0087] Step S202 performs absolute orientation, determining the position and orientation of camera 108 in the absolute coordinate system. However, errors are present at this stage. Therefore, to reduce these errors, the processes in steps S102 and S203 and S204 are performed.

[0088] If the convergence of the adjustment calculation results in step S202 exceeds a predetermined convergence range, a notification will be issued prompting the execution of the process related to step S101 again. In the execution of step 101 again, the slewing body 103 will be rotated after moving to a flatter location, and the slewing body 103 will be rotated with changes in the rotation angle and rotation speed. For example, by changing the rotation conditions of the slewing body 103, the acquisition of the initial values ​​described above will change, and the accuracy of the adjustment calculation may improve.

[0089] In step S102, the rotating body 103 is kept stationary while the traveling body 102 moves in a straight line. This straight line movement can be either forward or backward. It can also be forward followed by backward movement, or backward movement followed by forward movement. The distance traveled must be 0.5m or more. The repeated measurement of the position of the reflecting prism 107 by the total station 300 and the repeated photography by the camera 108 are continued in step S102 as well.

[0090] After step S102, a three-dimensional reconstruction process is performed based on the image data obtained by camera 108 in step S102 (Figure 4: step S203). In this process, mutual localization is performed to determine the relative relationship between the position and orientation of camera 108 at the time of stereo image capture and the feature points in the stereo image, and a three-dimensional model showing these relationships is obtained.

[0091] In step S203, the three-dimensional model obtained during the straight-line motion in step S102 is added (combined) to the three-dimensional model obtained by the adjustment calculation in step S202. The three-dimensional model obtained in step S101 and the three-dimensional model obtained in step S102 overlap. This is because the shooting ranges of camera 108 overlap.

[0092] Next, a re-adjustment calculation is performed using equations 1 and 2 related to the three-dimensional model obtained in step S203 (step S204). In step S102, the reflective prism 107 and the camera 108 move parallel to each other along a straight line. Therefore, the scale (absolute value of distance information) is accurately given, improving the accuracy of the position information in the adjustment calculation in step S204. In addition, the three-dimensional model adjusted in step S202 (the three-dimensional model obtained when the rotating body 103 rotates) is integrated with the three-dimensional model obtained in step S203 and the re-adjustment calculation is performed simultaneously, so any remaining errors that could not be adjusted in step S202 are reduced.

[0093] In step S204, L' in equation 2 is determined exactly. In step S204, instead of performing an adjustment calculation, a method may be used in which the amount of movement of the reflective prism 107 and the camera 108 in the straight-line portion is compared, and the amount of movement of the camera 108 is adjusted to match the amount of movement of the reflective prism 107. If the convergence of the result of the adjustment calculation in step S203 exceeds a predetermined convergence range, a notification is given prompting the execution of the process related to step S102 again. In the execution of step 102 again, one or more of the following are performed: moving to a flatter location and moving the rotating body 103 in a straight line, changing the distance of movement, and changing the acceleration and deceleration during straight-line movement.

[0094] In step 102, the reflective prism 107 and the camera 108 are moving in parallel lines. Therefore, the offset amount between the position of the camera 108 and the position of the reflective prism 107 can be accurately determined by the adjustment calculation in step S204 using equations 1 and 2.

[0095] Next, the cutting edge 106a or bucket 106 is used to push against the ground, lifting the heavy machinery 100 (lifting the front of the slewing body 103 off the ground) and tilting the slewing body 103 (step S103). The tilting angle is set to 5° from the horizontal plane to an angle at which the heavy machinery 100 does not tip over. Then, the three-dimensional data based on the image data obtained by the camera 108 during this process is added to the three-dimensional model obtained by the adjustment calculation in step S204, and the integrated three-dimensional model is restored (step S205).

[0096] The three-dimensional model obtained by the adjustment calculation in step S204 and the three-dimensional model obtained in step S103 overlap. This is because the objects captured by camera 108 in steps S101, S102, and S103 overlap. Therefore, the three-dimensional model obtained at this stage (step S205), which integrates the three-dimensional models obtained in steps S101 to S103, is an integration of the three-dimensional models obtained for each of the different movements of the rotating body 103.

[0097] In step S205, once the integrated three-dimensional model described above is obtained, adjustment calculations using equations 1 and 2 are performed on it (step S206). In step S205, three-dimensional data is obtained when a tilt occurs in the vertical direction, and this is used to perform the adjustment calculations (optimization calculations for each parameter) in step S206.

[0098] The process up to step 204 involved optimizing unknown parameters using three-dimensional data acquired during movement in the horizontal plane. By performing adjustment calculations that include three-dimensional data when the rotating body 103 is tilted, adjustment calculations that take into account changes in three-dimensional data in the vertical direction can be performed, further improving the accuracy of the optimized unknown parameters.

[0099] Furthermore, in step S103, the front of the rotating body 103 is lifted, and the rotating body 103 tilts backward. As a result, information is obtained that associates the positional relationship between the reflective prism 107, the rotation center position of the rotating body 103, and the camera 108 with the orientation of the rotating body 103.

[0100] If the convergence of the adjustment calculation results in step S206 exceeds a predetermined convergence range, a notification is issued prompting the execution of the process related to step S103 again. In the execution of step 103 again, the lifting operation is performed again with changes to the lifting (tilting) angle and speed of the aircraft.

[0101] The processing in steps S201 to S206 determines the relationship between the position of the reflecting prism 107, the position of the camera 108, and the rotation center position of the rotating body 103, as well as the relationship between this relationship and the orientation of the rotating body 103. The information obtained here becomes the initial calibration value obtained through the initial calibration.

[0102] By obtaining initial calibration values, and given the positions of the reflective prism 107 and the rotating body 103 in the absolute coordinate system, information on the position and orientation of the camera 108 in the absolute coordinate system can be obtained.

[0103] (Actual operation) Perform initial calibration in advance. It is also possible to perform initial calibration at the work site. Then, before starting work, set up the total station 300's instrument point. Select an instrument point at a location where the position of the reflecting prism 107 can be measured.

[0104] After the machine is set up, the total station 300 starts measuring the position of the reflecting prism 107 and the camera 108 starts taking images. The interval (repetition frequency) of the total station 300 measuring the position of the reflecting prism 107 and the interval (repetition frequency) of the camera 108 taking images may be the same or different. In this state, on-site calibration is performed, and then work with the heavy machinery 100 begins. As work with the heavy machinery 100 begins, processing related to the position and orientation of the cutting edge 106a starts.

[0105] The rotation center position of the pivoting body 103 in the absolute coordinate system is determined by the on-site calibration process. At this point, the position of the reflecting prism 107 in the absolute coordinate system is measured by the total station 300. From this measured position, the rotation center position of the pivoting body 103, and the measurements from the tilt sensors 113 to 116, the position and orientation of the cutting edge 106 in the absolute coordinate system are calculated.

[0106] As vehicle 100 moves, the rotation center position of the slewing body 103 also moves. This movement is tracked by SfM based on images captured by camera 108. The relationship between the position of the reflective prism 107, the rotation center position of the slewing body 103, and the position and orientation of camera 108 is known information from initial calibration. Therefore, the position and orientation of camera 108 in the absolute coordinate system at that time can be determined by field calibration.

[0107] When the mobile unit 101 moves from the field calibration state, the reflective prism 107 and camera 108 also move. Here, the movement of the reflective prism 107 is tracked by the total station 300 and its position in the absolute coordinate system is measured. Similarly, the movement of the camera 108 is tracked by SfM and its position in the absolute coordinate system is measured. Here, the relationship between the position of the reflective prism 107, the rotation center position of the rotating body 103, and the position of the camera 108 is known information from the initial calibration.

[0108] Therefore, the rotation center position of the slewing body 103 in the absolute coordinate system after the vehicle 101 has moved can be determined. Obtaining the rotation center position of the slewing body 103 in the absolute coordinate system makes it possible to calculate the position and orientation of the cutting edge 106a by measuring the position of the reflecting prism 107 with the total station 300. In other words, even if the vehicle 100 moves after field calibration, it is possible to obtain information on the position and orientation of the cutting edge 106a. This is the same even if the movement is intermittent, and field calibration only needs to be performed once before work.

[0109] (Superiority) Even if the traveling body 101 moves and the rotation center position of the rotating body 103 shifts, a new rotation center position can be determined using SfM with images captured by the camera 108. Therefore, there is no need to rotate the rotating body 103 again to determine the rotation center after the movement. This allows for increased work efficiency of the heavy machinery 100.

[0110] Precise positional data for the reflective prism 107 and camera 108 is obtained during the initial calibration process. Therefore, precision in the installation position of the reflective prism 107 and camera 108 is not required, and no complicated work is necessary.

[0111] During the initial calibration, three-dimensional data is acquired for actions such as the rotation of the rotating body 103, the linear movement of the traveling body 101, and intentionally tilting the rotating body 103, and initial values ​​necessary for SfM are obtained based on images captured by the camera 108. This process allows for the acquisition of information on the positional relationship between the camera 108 and the reflective prism 107, which are positioned on the rotating body 103, in a simple and highly accurate manner.

[0112] Furthermore, by performing the initial calibration in three stages, the accuracy of obtaining unknown parameters obtained through adjustment calculations can be improved.

[0113] In other words, in the first stage (step S101), adjustment calculations are performed using three-dimensional data obtained from the rotation of the rotating body 103. This allows for adjustment calculations related to the positional relationship between the reflective prism 107 and the camera 108, including the rotation center position of the rotating body 103.

[0114] In the second stage (step S102), the rotating body 103 is not rotated, and adjustment calculations are performed using the three-dimensional data obtained while the traveling body 101 is moving in a straight line. At this time, the reflective prism 107 and the camera 108 move in parallel in a straight line, so an accurate scale is given to the three-dimensional relationship between the reflective prism 107 and the camera 108 during the adjustment calculations.

[0115] In the third stage (step S103), the rotating body 103 is tilted, and adjustment calculations are performed using three-dimensional data in the vertical direction (Z-axis direction). This improves the accuracy of each parameter in the vertical plane, in addition to the accuracy of each parameter in the horizontal plane.

[0116] 2. Second Embodiment Figure 6 shows an overview of this embodiment. This embodiment relates to a technique for more easily obtaining the preliminary data necessary for calculating the blade tip 106a by utilizing stereo photometry. In this embodiment, cameras 121 and 122, which serve as stereo cameras, are prepared. Cameras 121 and 122 are mounted on the swivel body 103 at a distance from each other so that the blade tip 106a portion is within the shooting range. Stereo photography of the blade tip 106a portion is performed by cameras 121 and 122.

[0117] Here, the initial position and orientation of cameras 121 and 122 in the absolute coordinate system are determined using the technology of the first embodiment. This enables stereo photographic measurement of the blade tip 106a using cameras 121 and 122, and the position and orientation of the blade tip 106a relative to the rotation center position of the rotating body 103 are determined. At this time, the measurement position of the reflection prism by the total station 300 and the measurement values ​​of the tilt sensors 113 to 116 are obtained. By performing the above operation multiple times while changing the position of the blade tip 106a, the necessary preliminary data for calculating the position and orientation of the blade tip 106a is obtained.

[0118] Since cameras 121 and 122 capture the area of ​​the cutting edge 106a, image recordings and three-dimensional information of the work performed by the heavy machinery 100 can also be obtained. It is also possible to use a configuration with three or more cameras.

[0119] According to this embodiment, the process of acquiring the necessary preliminary data for calculating the position and orientation of the cutting edge 106a can be easily performed. For example, the system can be made operational by performing the following steps. (1) Attachment of the reflective prism 107, camera 121, and camera 122 to the rotating body 103 (2) Installation of tilt sensors 113-116 (3) Initial calibration process (4) Preliminary data for calculating the position and orientation of the cutting edge 106a is acquired using stereo photogrammetry by cameras 121 and 122. (5) Field calibration process

[0120] In this embodiment, as in the first embodiment, after on-site calibration processing is performed at the start of work, there is no need to interrupt the work even if the traveling body 101 moves, and the position and orientation of the cutting edge 106a are acquired.

[0121] In stereo photogrammetry, the precise positions and orientations of the two cameras constituting the stereo camera must be determined, which involves complicated preliminary work. In this embodiment, the camera positions and orientations are calculated during the initial calibration process, so the installation positions of cameras 121 and 122 only need to be approximate. Therefore, complicated work is not required, and high practicality is achieved. This is also true when using three or more cameras.

[0122] In this embodiment, the position and orientation of the blade tip 106a can be directly determined by performing three-dimensional photometry of the blade tip using multiple cameras. In this case, the position and orientation of each camera used in the absolute coordinate system can be determined from the results of the initial calibration and the measurement results of the position of the reflective prism 107 by the total station 300. Then, the position and orientation of the blade tip 106a in the absolute coordinate system can be determined by three-dimensional photometry. In this case, information on the position and orientation of the blade tip 106a can be obtained without using the tilt sensors 114-116 (of course, tilt sensors 114-116 may also be used). In this case as well, the simplicity of not requiring high precision in the position and orientation of the multiple cameras used during installation can be obtained.

[0123] 3. Third Embodiment The present invention can also be used in a technique for calculating the cutting edge position by installing a GNSS (Global Navigation Satellite System) antenna on the rotating body of heavy machinery and measuring the position of the antenna in an absolute coordinate system. When measuring the position using GNSS, it is preferable to use relative positioning, which has high measurement position accuracy.

[0124] In this case, a GNSS receiving antenna is installed on the rotating body 103 instead of the reflecting prism 107. The system then operates when the antenna position is measured by GNSS. In this case, a GNSS antenna is attached to the rotating body 103. The position of the GNSS antenna in the absolute coordinate system is measured by GNSS, and the position and orientation of the cutting edge 106a are calculated using these measurements. Except for using GNSS as position information, this is the same as the first embodiment. This embodiment can also be applied to the second embodiment.

[0125] 4. Fourth Embodiment Figure 7 is a flowchart for processing the three-dimensional data obtained in steps S101-S103 of Figure 3 together. In this case, three-dimensional reconstruction (mutual orientation) is performed using the data obtained in steps S101-S103 of Figure 3 (step S301), and then adjustment calculations are performed using the position information of the reflecting prism 107 (step S302). Here, adjustment calculations are performed using equations 1 and 2.

[0126] 5. Others This invention can be applied to heavy machinery other than hydraulic shovels, as long as it is equipped with a traveling body and a rotating body. This invention can also be applied to heavy machinery that operates unmanned.

[0127] When step S101 is executed, the process of S201 and the process of the subsequent step S202 may be performed in parallel. Also, when step S102 is executed, the process of S203 and the process of the subsequent step S204 may be performed in parallel. Also, when step S103 is executed, the process of S205 and the process of the subsequent step S206 may be performed in parallel. After the completion of the process in Figure 3, the process in Figure 4 may be executed. [Explanation of Symbols]

[0128] 100...Heavy machinery, 101...Motorized body, 102...Continuous track, 103...Slewing body, 104...Boom, 105...Arm, 106...Bucket, 106a...Cutting edge, 107...Reflective prism, 108...Camera, 113...Tilt sensor, 114...Tilt sensor, 115...Tilt sensor, 116...Tilt sensor, 121...Camera, 122...Camera, 200...Calculation unit, 300...Total station.

Claims

1. A vehicle equipped with means of transport, A rotating body that can rotate on the aforementioned traveling body and is equipped with an optical target and a camera, A method for obtaining information on heavy machinery having, In the process of rotating the rotating body, a first step is to measure the position of the optical target using a surveying device and perform SfM using the image captured by the camera, In the process of moving the traveling body in a straight line without rotating the rotating body, a second step is to measure the position of the optical target using the surveying device and perform SfM using the image captured by the camera, A method for acquiring information on heavy machinery, wherein the relationship between the position of the optical target and the position of the camera on the rotating body is determined based on the position of the optical target obtained by the surveying device in the first step and the second step, and the position and orientation of the camera obtained by SfM.

2. A vehicle equipped with means of transport, A rotating body that can rotate on the aforementioned traveling body and is equipped with an optical target and a camera, A method for obtaining information on heavy machinery having, In the process of rotating the rotating body, a first step is to measure the position of the optical target using a surveying device and perform SfM using the image captured by the camera, In the process of moving the traveling body in a straight line without rotating the rotating body, a second step is to measure the position of the optical target using the surveying device and perform SfM using the image captured by the camera, In the process of tilting the rotating body from the horizontal, a third step is to measure the position of the optical target using the surveying device and perform SfM using the image captured by the camera, A method for acquiring information on heavy machinery, wherein the relationship between the position of the optical target and the position of the camera on the rotating body is determined based on the position of the optical target obtained by the surveying device and the position and orientation of the camera obtained by SfM, as obtained in the first to third steps.

3. In the first and second steps, bundle adjustment calculations are performed based on the images captured by the camera. The aforementioned bundle adjustment calculation is performed as follows: A method for acquiring information on heavy machinery according to claim 1 or 2, wherein the constraint condition is that the difference between the position of the optical target obtained by the surveying device in the first step and the second step and the position of the optical target obtained by SfM is minimized.

4. The aforementioned heavy machinery is a hydraulic shovel, The method for acquiring information on heavy machinery according to claim 2, wherein in the third step, the rotating body is tilted from the horizontal direction by pushing the ground with the bucket provided on the hydraulic shovel.

5. A vehicle equipped with means of transport, A rotating body that can rotate on the aforementioned traveling body and is equipped with an optical target and a camera, A device for handling information on heavy machinery, In the process of rotating the rotating body, a first step is to measure the position of the optical target using a surveying device and perform SfM using the image captured by the camera, In the process of moving the vehicle in a straight line without rotating the rotating body, a second step is to measure the position of the optical target using the surveying device and perform SfM using the image captured by the camera. Execute, A device for handling information about heavy machinery, comprising a calculation unit that determines the relationship between the position of the optical target and the position of the camera on the rotating body, based on the position of the optical target obtained by the surveying device in the first step and the second step, and the position and orientation of the camera obtained by SfM.

6. A vehicle equipped with means of transport, A rotating body that can rotate on the aforementioned traveling body and is equipped with an optical target and a camera, A program that causes a computer to process information about heavy machinery having the following characteristics: to the computer In the process of rotating the aforementioned rotating body, a first step is to measure the position of the optical target using a surveying device and perform SfM using the image captured by the camera, In the process of moving the vehicle in a straight line without rotating the rotating body, the second step involves measuring the position of the optical target with the surveying device and performing SfM using the image captured by the camera. Make it run, A program that determines the relationship between the position of the optical target and the position of the camera on the rotating body, based on the position of the optical target obtained by the surveying device in the first step and the second step, and the position and orientation of the camera obtained by SfM.

Images