Calibration method, calibration device, and construction machine

Abstract

The purpose of the present disclosure is to provide a method and system for calibrating a position and a posture of an imaging device even when the imaging device installed in a construction machine has a limited range for acquiring an image of a bucket. A calibration method according to the present disclosure, in a construction machine in which a rotating body is mounted on a traveling body, acquires an image using an imaging device attached to the rotating body, acquires coordinates of a reference point within the image, the reference point being defined by a predetermined portion of the construction machine, rotates the rotating body with respect to the traveling body, and calibrates the imaging device using information of the reference point (see figure 8).

Description

Calibration method, calibration device, and construction machinery

[0001] The present disclosure relates to a technique for calibrating an imaging device provided in a construction machine.

[0002] Patent Literature 1, for example, describes a method for determining the position and orientation of a bucket of a construction machine (excavator), in which an image capturing device (camera) mounted on the vehicle body acquires a bucket image including a preset marker attached to the bucket, thereby determining the position and orientation information of the image capturing device relative to the bucket, and converting the position and orientation information of the camera relative to the bucket into position and orientation information of the bucket relative to the vehicle body. This makes it possible to calibrate the position and orientation of the image capturing device.

[0003] Japanese Patent Application Laid-Open No. 2020-140696

[0004] In the above-mentioned Patent Document 1, the imaging device acquires an image of the bucket, thereby determining information about the position and orientation of the imaging device. However, depending on the installation position of the imaging device, the range in which the image of the bucket can be acquired may be limited, or it may not be possible to acquire an image of the bucket. In such cases, it is not possible to determine information about the position and orientation of the imaging device.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide a method and device for calibrating the position and attitude of an imaging device installed on a construction machine, even when the range in which bucket images can be acquired is limited.

[0006] The calibration method disclosed herein involves a construction machine having a rotating body mounted on a running body, acquiring an image using an imaging device attached to the rotating body, obtaining the coordinates of a reference point in the image using a predetermined location on the construction machine as a reference point, rotating the rotating body relative to the running body, and calibrating the imaging device using the information on the reference point.

[0007] According to the calibration method of the present disclosure, the position and attitude of the imaging device can be calibrated even when the imaging device installed on a construction machine has a limited range in which it can acquire images of the bucket.

[0008] 1 is a diagram illustrating the configuration of a hydraulic excavator 100 according to a first embodiment. FIG. 1 is a top view of a revolving body 102. FIG. 2 shows an example configuration of a control device 103. FIG. 2 is a diagram illustrating a method for estimating the position and attitude of an imaging device S1. FIG. 3 shows an example of a location where the imaging device S1 is attached to the hydraulic excavator 100. FIG. 4 shows an example of a location where the imaging device S1 is attached to the hydraulic excavator 100. FIG. 5 shows an example of an image acquired by the imaging device S1 displayed on a display 156 when the imaging device S1 is attached as described in FIGS. 5 and 6 . FIG. 6 is a flowchart illustrating a procedure for calibrating the position and attitude of the imaging device S1. FIG. 7 shows an example of a table for storing data required for configuring the position and attitude of the imaging device S1. FIG. 8 shows an example of a data table for storing the position and attitude of the imaging device S1. FIG. 9 shows a diagram illustrating a preferred example of a reference point. FIG. 10 shows an example of a case where a reference point arranged on a traveling body and a reference point arranged on a revolving body are combined. FIG. 11 is a flowchart illustrating a procedure for calibrating the imaging device S1 by combining a reference point arranged on a traveling body and a reference point arranged on a revolving body. 17 shows an example of acquiring a reference point when a preset turning angle is reached. FIG. 17 shows an example of superimposing support information on an image captured by the imaging device S1 and displaying it on the display 156. FIG. 17 shows a flowchart explaining the procedure for superimposing support information on an image captured by the imaging device S1 and displaying it on the display 156. FIG. 17 shows an example of placing a marker 1701 on the side of the running body 101 to automatically detect a reference point. FIG. 17 shows an example of a flowchart explaining the procedure for automatically detecting a shift in the position or attitude of the imaging device S1 by automatically detecting a shift in the position of the reference point. FIG. 17 shows an example of detecting a shift in the position or attitude of the imaging device S1 using the trajectory of the reference point.

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be designated by the same numerals. Note that the accompanying drawings show embodiments and implementation examples according to the principles of the present disclosure, but these are for understanding the present disclosure and are not to be used to interpret the present disclosure in a limiting manner. The descriptions in this specification are merely typical examples and are not intended to limit the scope or application of the present disclosure in any way.

[0010] In the following embodiments, the present disclosure is described in sufficient detail for those skilled in the art to implement the present disclosure, but it should be understood that other implementations and forms are possible, and that changes in configuration and structure and substitutions of various elements are possible without departing from the scope and spirit of the technical ideas of the present disclosure. Therefore, the following description should not be interpreted as being limited thereto.

[0011] <First Embodiment> Figure 1 is a diagram illustrating the configuration of a hydraulic excavator 100 according to a first embodiment of the present disclosure as one type of construction machine. The hydraulic excavator 100 is, as an example, a construction machine having a backhoe-type front structure in which a bucket 113 faces the operator compartment OR. Operation of the hydraulic excavator 100 excavates ground L, which is an excavation target, such as a plane L0, a plane L1, and a slope L2. The soil excavated by the hydraulic excavator 100 is discharged (released) into a vessel 200.

[0012] As an example, the hydraulic excavator 100 has a lower traveling body (hereinafter simply referred to as the traveling body) 101, and an upper rotating body (hereinafter simply referred to as the rotating body) 102 that is rotatably supported on the traveling body 101. The rotating body 102 is also provided with a cab OR that forms an operation room where an operator sits, a control device 103, and various sensors (an imaging device S1, a surrounding sensor S2, and an attitude sensor S3).

[0013] The hydraulic excavator 100 also has a front working mechanism (hereinafter simply referred to as the working mechanism) 130 consisting of a boom 111 attached to the rotating body 102, an arm 112 attached to the tip of the boom 111, and a bucket 113 attached to the tip of the arm 112. The hydraulic excavator 100 performs excavation work on the ground L by driving the working mechanism 130 (boom 111, arm 112, bucket 113) by an engine (power source), hydraulic cylinder (actuator), etc. (not shown).

[0014] The running body 101 has a frame and a track wound around the frame, and is a component that causes the hydraulic excavator 100 to travel when the track is rotated by a travel motor. The rotating body 102 rotates 360 degrees (pivoting operation) relative to the running body 101. The direction, speed, and angle of the pivoting operation are controlled by, for example, an operator.

[0015] A control device 103 is mounted on the revolving unit 102. The control device 103 acquires images using an imaging device S1 including an image sensor, and generates support information including the current positions of the bucket 113, the traveling body 101, the work ground surface, and the vessel 200 such as a dump truck, based on sensor information obtained from a surrounding sensor S2 and an attitude sensor S3.

[0016] FIG. 2 is a top view of the revolving unit 102. The revolving unit 102 is provided with an imaging device S1, a surrounding sensor S2, and an attitude sensor S3 for detecting the situation around the hydraulic excavator 100 and the attitude of the hydraulic excavator 100. As shown in FIG. 2 , as an example, the imaging device S1 is attached to the left side of the revolving unit 102 of the hydraulic excavator 100 at a position protruding from the left side of the building of the revolving unit 102. The imaging device S1 has an imaging surface (receiving surface or light-receiving surface) such as a CMOS sensor oriented to capture an image of a field of view SR1 looking forward (i.e., forward left) from the left side of the revolving unit 102. In this embodiment, the imaging device S1 is configured to acquire an image including an area including at least a portion of the traveling unit 101 and the planned traveling path of the traveling unit 101.

[0017] The surrounding sensor S2 is an object detection sensor such as a LiDAR (Light Detection and Ranging) or ultrasonic sensor that detects the presence or absence of an object and its distance within the field of view SR2. The surrounding sensor S2 is not limited to this and may be composed of multiple LiDAR or ultrasonic sensors. Furthermore, the imaging device S1 and the surrounding sensor S2 may be composed of a single or multiple stereo cameras capable of imaging and stereoscopic vision. The attitude sensor S3 is a sensor that detects the attitude of the hydraulic excavator 100 and is capable of acquiring the attitudes of the boom 111, arm 112, and bucket 113 and the rotation angle of the rotating unit 102 relative to the traveling unit 101. The attitude sensor S3 may be composed of, for example, a gyro sensor, an inclination sensor, an acceleration sensor, or a combination of these. Furthermore, the example illustrated in FIG. 2 is merely an example, and the present disclosure is not limited thereto.

[0018] 2, in addition to the imaging device S1 that captures an image of the left front field of view SR1 described above, an imaging device S11 that captures an image of the right front field of view SR11 and an imaging device S12 that captures an image of the front field of view SR12 are attached to the rotating unit 102. For example, the hydraulic excavator 100 includes multiple cameras corresponding to the imaging devices S1, S11, and S12.

[0019] For example, in this embodiment, the imaging device S11 is attached to the right side of the revolving unit 102 at a position protruding from the right side of the building of the revolving unit 102, and is configured to capture an image of the area to the right front of the running unit 101, including at least a portion of the running unit 101. In addition, the imaging device S12 is attached to the front of the revolving unit 102 (the surface to which the work device 130 is attached; in this embodiment, the lower part of the cab OR) at a position protruding from the front of the building of the revolving unit 102, and is configured to capture an image of the area in front of the running unit 101, including at least a portion of the running unit 101 and at least a portion of the work device 130.

[0020] The imaging devices S1, S11, and S12 are not limited to having the above-described configuration. For example, the imaging areas of the imaging devices S1, S11, and S12 are not limited to the area viewed in three directions from the running body 101. For example, imaging devices that capture images of the entire surroundings may be provided by adjusting the installation position, angle, and number of imaging devices. For example, in addition to the imaging devices S1, S11, and S12, an imaging device configured and arranged to capture an area behind the running body 101, including at least a portion of the running body 101, may be provided on the rear surface of the building of the rotating body 102 or at a position toward the rear of the center of the bottom surface. By adding this rear imaging device, it is possible to capture images of both the areas in front and behind the running body 101. This makes it possible for the imaging device to capture an image including a portion of the running body 101 and the area along its travel path, regardless of whether the running body 101 is moving forward or backward.

[0021] In other words, in this embodiment, each of these imaging devices needs only to be positioned on the rotating body 102 in a position and attitude such that it images an area including at least a portion of the running body 101 and the planned running path of the running body 101.

[0022] 3 shows an example configuration of the control device 103. As an example, the control device 103 includes a reference point coordinate acquisition unit 1031, a reference point coordinate calculation unit 1032, a calibration unit 1033, a support information generation unit 1034, and a drive control unit 1035. The control device 103 includes, for example, a CPU 151 (processor), a ROM 152, a RAM 153, and a hard disk drive 154. An input device 155 and a display 156 (display unit) are also provided in the cab OR of the revolving unit 102. The control device 103 performs information processing based on information from the input device 155 and various sensors, outputs the results to the display 156, and controls the operation of the hydraulic excavator 100. The CPU 151 is one aspect of a processor and may be a GPU, a semiconductor device capable of other arithmetic processing, or a combination thereof.

[0023] The reference point coordinate acquisition unit 1031 acquires the coordinates of the reference point required to calibrate the position and attitude of the imaging device S1 in the image acquired from the imaging device S1. To acquire the coordinates of the reference point, as an example, the imaging device S1 controls the attitude of the construction machine 100 so that it includes the tip of the traveling body 101, and the user selects the tip of the crawler on the image displayed on the display 156 by touch operation or cursor operation (not shown). This makes it possible to acquire the coordinates of the reference point in the image.

[0024] The reference point coordinate calculation unit 1032 calculates the position of the tip of the crawler in three-dimensional space based on the three-dimensional information acquired from the surrounding sensor S2 and the attitude information acquired from the attitude sensor S3.

[0025] The calibration unit 1033 estimates the position and attitude of the imaging device S1 using the coordinates on the display of the tip of the crawler and the information on the position in three-dimensional space acquired above. At this time, since multiple pieces of information that serve as reference points are required as will be described later, the drive control unit 1035 drives the rotation drive unit D1 to change the rotation angle of the rotating body relative to the running body, change the position of the reference point on the display, and acquire the information on the reference point again.

[0026] The support information generation unit 1034 generates support information to be superimposed on the image acquired by the imaging device S1 from the information on the position and attitude of the imaging device S1 estimated by the calibration unit 1033 and the information on the attitude of the hydraulic excavator 100 acquired by the attitude sensor S3, and outputs this support information to the display 156 while superimposing it on the image. For example, when the traveling object 101 is captured as an image by the imaging device S1, the support information generation unit 1034 generates, as support information, guide lines indicating the traveling direction of the traveling object 101 using the turning angle information acquired by the attitude sensor S3. By outputting this support information to the display 156 while superimposed on the image acquired by the imaging device S1, the occupant can easily confirm the traveling direction of the hydraulic excavator 100.

[0027] FIG. 4 is a diagram illustrating a method for estimating the position and orientation of the image capture device S1. Here, the image capture device S1 is a general camera, and the internal parameter K of the camera is calculated based on a known focal length (f x, f y ) and the principal point position ( c x , c y ) is used to express it as Equation 1. It is assumed that the distortion of the camera lens has been corrected, and the sample points in the world coordinate system are known using the three-dimensional information and posture information of the vehicle model of the hydraulic excavator 100.

[0028]

[0029] The external parameters of the camera, which are the information we want to obtain, are extrinsic Assuming that the orientation R and the position t → The combination of T extrinsic The transformation matrix from world coordinates to camera coordinates is T projection Then, the relationship of Equation 2 is obtained.

[0030]

[0031] The sample point in the world coordinate system is r → , the transformation matrix from world coordinates to camera coordinates is T projection Then, the sample point p within the field of view captured by the camera → is expressed by Equation 3. λ is a coefficient used when converting from world coordinates to camera coordinates.

[0032]

[0033] From Equation 3, three equations are generated for each sample point, but the unknown constant λ increases by one. projection is expressed as a combination of a 3x3 matrix representing the orientation and a 3x1 matrix representing the position, so there are a total of 12 unknown parameters. However, since the equation is equivalent even if multiplied by a constant, the number of unknowns becomes 11. Therefore, if n samples are taken, the relationship of Equation 4 holds, and if six or more samples are taken, T projection is obtained, and it can be seen that the orientation and position of the camera can be estimated using Equation 2. The above is generally called the PnP (Perspective n Point) problem.

[0034]

[0035] 5 and 6 show an example of where the imaging device S1 is attached to the hydraulic excavator 100. Fig. 5 shows the hydraulic excavator 100 as seen from the side, and Fig. 6 shows the hydraulic excavator 100 as seen from above. In this example, in order to easily confirm the direction in which the traveling body 101 is traveling while the hydraulic excavator 100 is traveling, the imaging device S1 is attached at a position lower than the operation cabin OR of the rotating body 102 so as to provide a field of view in front of the rotating body, and is also attached outward of the traveling body 101 as seen from above. This makes it possible to confirm the tip of the traveling body 101 when the rotating body is facing forward or backward relative to the traveling body.

[0036] As will be described later, support information can be superimposed on the image captured by the imaging device S1 and displayed on the display 156. If information about the position and attitude of the imaging device S1 is not properly retained at this time, the state of the hydraulic excavator 100 displayed on the display 156 cannot be properly calculated, resulting in a discrepancy in the display of the support information. Therefore, a calibration method is required to properly obtain information about the position and attitude of the imaging device S1 when the imaging device S1 is attached to the hydraulic excavator 100. Even after calibration, a discrepancy in the position or attitude of the imaging device S1 may occur due to unintentional contact or the like. In such cases, it is desirable to have a means for recalibration or for automatically detecting the discrepancy. A display prompting calibration may be displayed on the display 156 when the system is started for the first time after the imaging device S1 is attached to the hydraulic excavator 100, or a display prompting calibration may be displayed again on the display 156 when a discrepancy in the position or attitude of the imaging device S1 is detected. Furthermore, when a deviation in the position or orientation of the image capture device S1 is detected, calibration may be performed automatically as described below, and newly obtained information on the position and orientation of the image capture device S1 may be used to superimpose support information and display it on the display 156. Furthermore, an operation mode for performing calibration may be provided, and the user may actively perform calibration.

[0037] 7 shows an example of an image captured by the imaging device S1 displayed on the display 156 when the imaging device S1 is installed as described in FIGS. 5 and 6 . The front portion of the boom 111, arm 112, and bucket 113 are hidden by the rotating unit 102 and cannot be seen from the imaging device S1. Therefore, it is not possible to calibrate the position and attitude of the imaging device S1 using, for example, the tip of the bucket 113 as a reference point. For example, if the front portion is only visible within 20% of the angle of view of the imaging device S1, it is difficult to calibrate the imaging device using a part of the front portion as a reference point. Therefore, calibration is performed by setting, for example, the tip of the traveling unit 101 as a reference point 701.

[0038] FIG. 8 is a flowchart illustrating a procedure for calibrating the position and orientation of the image capturing device S1. In step S801, the image capturing device S1 acquires an image and outputs the image on the display 156. In step S802, the user uses the input device 155 to operate the cursor on the display 156 and click a reference point on the image output on the display 156. The reference point coordinate acquisition unit 1031 acquires the clicked coordinate as the coordinate of the reference point in the image. In step S803, the orientation sensor S3 acquires the orientation of the shovel and calculates world coordinates corresponding to the selected reference point. In step S804, the reference point coordinate acquisition unit 1031 determines whether the required number of reference points have been acquired. As described above, information on at least six reference points is required. However, since the greater the number of reference points, the higher the calibration accuracy, a greater number of reference points may be set as the threshold. If the required number of reference points has not been acquired in step S804, the drive control unit 1035 rotates the rotating body relative to the running body via the rotation drive unit D1 in step S805, and then repeats steps S801 to S804. The rotation in step S805 only needs to be performed within a range in which the reference points are expected to be within the display; for this determination, initial values ​​may be assigned to the position and attitude of the camera, and the rotating body may be rotated within the expected range based on these initial values. If the required number of reference points has been acquired in step S804, the calibration unit 1033 calibrates the imaging device S1 in step S806.

[0039] 9 shows an example of a table for storing data necessary for determining the position and orientation of the image capture device S1. This data table can be stored in the hard disk drive 154. This data table holds the camera coordinates acquired in step S802 and the world coordinates calculated in step S803 for the number n of necessary reference points. Based on this, the calibration unit 1033 calibrates the image capture device S1.

[0040] 10 shows an example of a data table for storing the position and orientation of the image capture device S1. This data table can be stored in the hard disk drive 154. This data table stores a total of six parameters: X, Y, and Z for position, and θp, θy, and θr for orientation, which correspond to pitch angle, yaw angle, and roll angle. The data table may be pre-stored with initial values ​​corresponding to the mounting position and orientation of the image capture device S1, or the estimated results of the position and orientation obtained as a result of calibration may be added to or overwritten.

[0041] FIG. 11 is a diagram showing a preferred example of the reference point 1101. If the tip (reference point 701) in FIG. 7 is the reference point, the angle between the normal to the plane on which the reference point 1101 is located (in this example, the side of the vehicle 101) and the optical axis of the imaging device S1 is nearly a right angle, making it difficult to accurately select the reference point 1101 on the image on the display 156. In this case, the accuracy of the position and orientation of the imaging device S1 obtained by calibration may also be poor. In FIG. 11 , the imaging device S1 attached to the left side of the rotating unit captures an image of the tip of the vehicle 101 on the right side of the running unit. In this case, the angle between the normal to the side of the vehicle 101, including the reference point 1101, and the optical axis of the imaging device S1 is nearly parallel, making it possible to accurately select the reference point 1101 on the image on the display 156. Therefore, the accuracy of the position and orientation of the imaging device S1 obtained by calibration is also good.

[0042] As an example, calibration accuracy can be improved by acquiring a reference point at a rotation angle such that the angular difference between the normal to the plane containing the reference point and the optical axis of the image capture device S1 is 45° or less. Since six or more reference points must be acquired, the angular difference between the normal to at least one of the reference points and the optical axis of the image capture device S1 may be 45° or less, or all reference points may satisfy the same condition. Furthermore, it is not necessary to acquire six or more reference points using only the reference point 1101 (i.e., only reference points on the vehicle 101). Reference points may be acquired by combining the reference point 701 and the reference point 1101, or a reference point may be placed at the rear end of the crawler. Furthermore, the reference point does not necessarily have to be at the tip of the vehicle 101; it may be anywhere on the vehicle that can be photographed by the image capture device S1.

[0043] 12 shows an example in which a reference point placed on the traveling body and a reference point placed on the revolving body are combined. Here, a case is shown in which part of the bucket 113 is visible to the imaging device S1 depending on the posture of the boom 111, arm 112, and bucket 113, and one of the toes of the bucket 113 is set as the reference point 1201. Also, the reference point does not necessarily have to be placed on the bucket 113; for example, a distinctive part such as a corner of the revolving body may be set as the reference point 1202. By combining the reference points placed on the revolving body and the reference points placed on the traveling body and acquiring information on six or more reference points, the imaging device S1 can be calibrated.

[0044] 13 is a flowchart illustrating the procedure for calibrating the image capture device S1 by combining a reference point placed on the running body and a reference point placed on the rotating body. S801, S802, S803, S805, and S806 are the same as those described in FIG. 8, and therefore will not be described again.

[0045] In step S1301, the reference point coordinate acquisition unit 1031 determines whether the number of reference points to be acquired on the predetermined traveling body has been acquired. If the required number of reference points on the traveling body has not been acquired, the process of acquiring the required reference points on the traveling body is repeated by changing the rotation angle in step S805. If the required number of reference points on the traveling body have been acquired, in step S1302 the reference point coordinate acquisition unit 1031 determines whether the required number of reference points have been acquired on the rotating body. If the required number of reference points have not been acquired on the rotating body, in step S1303 the drive control unit 1035 changes the position of the bucket via the front drive unit D2, and then repeats the processes of S801 to S1302. If the required number of reference points have been acquired on the rotating body, in step S806 the calibration unit 1033 calibrates the image capture device S1.

[0046] FIG. 14 shows an example in which a reference point is acquired when a preset rotation angle is reached. The rotation angle can be acquired by using a rotation angle sensor as the attitude sensor S3. However, depending on the sensor's specifications and configuration, errors may occur during rotation, and the sensor may be forcibly reset when a predetermined rotation angle is reached. If rotation angle information is acquired while an error is present, the reference point coordinate calculation unit 1032 will calculate the position of the reference point under that error, thereby degrading the accuracy of the calibration of the position and attitude of the image capture device S1 by the calibration unit 1033. Therefore, by pre-storing a rotation angle at which the rotation angle is forcibly reset and acquiring attitude information at such a rotation angle, it is possible to prevent accuracy degradation. Here, n rotation angles at which the sensor is reset are pre-set, and each rotation angle θi is distinguished by a counter i.

[0047] In step S1401, i is set to an initial value of 1. In step S1402, the drive control unit 1035 drives the rotation drive unit D1 so that the i-th rotation angle θi is obtained. Steps S801 to S804 are the same as those described in FIG. 8, and therefore description thereof will be omitted. If the required number of reference points has not been obtained in step S804, the counter i is incremented in step S1403, and the processing of steps S1402 to S804 is repeated. If the required number of reference points has been obtained in step S804, the calibration unit 1033 calibrates the imaging device S1 in step S806.

[0048] In Figure 14, resetting the sensor refers to a process of overwriting the turning angle detected by the sensor with a predetermined initial value. The initial value may be a different value for each turning angle at which the sensor is reset, or may be the same value for all turning angles. The process of resetting the sensor may be performed by the sensor itself, for example, or by any other computing device.

[0049] FIG. 15 shows an example in which support information is superimposed on an image captured by the imaging device S1 and displayed on the display 156. The traveling direction of the traveling body 101 is represented as support information 1501 by a series of arrow feathers with the tip pointing in the traveling direction of the hydraulic excavator 100, which are arranged along the edge of the surface of the traveling body 101 that is in contact with the ground. The support information 1501 allows the operator to easily confirm the traveling direction of the hydraulic excavator 100, thereby improving safety and reducing the burden on the operator. The operator may operate the hydraulic excavator 100 from inside the hydraulic excavator 100, or may transmit information about the hydraulic excavator 100 to a remote control room and operate the hydraulic excavator 100 while checking the display in the remote control room. In particular, when remotely operating the hydraulic excavator 100, information about the surrounding environment of the hydraulic excavator 100 is limited, so the superimposition of support information can significantly reduce the burden on the operator.

[0050] A supplementary note about the support information 1501: If the correct position and attitude of the image capturing device S1 are known, and the attitude of the excavator body is also known, the direction in which the traveling body 101 should be facing can also be known. Information instructing the hydraulic excavator 100 to proceed in that direction can be presented as the support information 1501. More specifically, if the position and attitude of the image capturing device S1 are known, T projection can be calculated, and this can be used to convert the three-dimensional coordinates of the support information into two-dimensional coordinates on the display 156. That is, accurate support information 1501 can be presented based on the calibration result of the imaging device S1.

[0051] 16 is a flowchart illustrating a procedure for superimposing support information on video captured by the imaging device S1 and displaying the superimposed support information on the display 156. In step S1601, the imaging device S1 acquires video. In step S1602, the attitude sensor S3 acquires the attitude of the shovel. In step S1603, the support information generation unit 1034 generates support information from the information about the attitude of the shovel acquired in step S1602 and the information about the position and attitude of the camera estimated by the calibration unit 1033. As described above, one example of the support information generated here is the support information 1501 shown in FIG. 15. In step S1604, the support information generated in step S1603 is superimposed on the video captured in step S1601 and displayed on the display 156.

[0052] <Summary of First Embodiment> The hydraulic excavator 100 according to the first embodiment acquires an image from an imaging device S1 attached to a rotating body of the hydraulic excavator 100, acquires the coordinates of a predetermined location on the hydraulic excavator 100 in the image as a reference point, rotates the rotating body, and calibrates the position and attitude of the imaging device S1 using the reference point coordinates in the image. By using the reference point on the hydraulic excavator 100 while rotating the rotating body, it is possible to calibrate the position and attitude of the imaging device S1 even if the range in which the imaging device S1 acquires images is limited.

[0053] Second Embodiment In the first embodiment, it has been described that the operator visually checks the reference point 701 or the reference point 1101 on the display 156 and selects the reference point using the input device 155. In a second embodiment of the present disclosure, a configuration example will be described in which the control device 103 automatically calculates the coordinates of the reference point on the display 156. The other configurations are the same as those in the first embodiment.

[0054] FIG. 17 shows an example in which a marker 1701 is placed on the side of the running body 101 to automatically detect a reference point. In the first embodiment, the tip of the running body 101 is used as the reference point. However, since a fixed position is preferable when automatically detecting a reference point, the marker 1701 is placed on a non-movable portion of the running body 101. The type of marker 1701 may be a checkered pattern as shown in FIG. 17, or a two-dimensional barcode or the like may be used to distinguish multiple reference points. By using such a distinctive pattern, the control device 103 can automatically calculate the position of the marker 1701 by image processing when the imaging device S1 captures the marker 1701. Furthermore, such image processing may calculate not only the coordinates of the marker 1701 but also a normal vector of the marker 1701, and use the normal vector to improve calibration accuracy.

[0055] As described above, in the processing flow of FIG. 8 , the reference point acquisition in step S802 can be performed automatically without user operation, thereby shortening the calibration time. Furthermore, by improving the accuracy of automatic detection of the marker 1701, it is also possible to calculate the position and orientation of the image capture device S1 with high accuracy. Furthermore, in the processing flow of FIG. 8 , the entire calibration process can be automated by automatically performing the rotation operation in step S805 by the drive control unit 1035. Furthermore, although an example of detecting the reference point using the marker 1701 has been shown here, it is also possible to extract characteristic locations in the appearance of the hydraulic excavator 100 by image processing and use them instead of the reference point without using the marker 1701.

[0056] 18 is an example of a flowchart illustrating a procedure for automatically detecting a shift in the position or orientation of the image capture device S1 by automatically detecting a shift in the position of the reference point. Here, it is assumed that calibration of the image capture device S1 has been completed in advance, and the estimated results of the position and orientation of the image capture device S1 have been stored in the table shown in FIG.

[0057] In step S1801, the image capturing device S1 acquires an image including the reference point. In step S1802, the control device 103 performs image processing to acquire the position of the marker 1701 as the camera coordinates of the reference point. In step S1803, the attitude sensor S3 acquires the attitude of the shovel. Furthermore, the reference point coordinate calculation unit 1032 calculates the camera coordinates corresponding to the reference point using information on the calibrated position and attitude of the image capturing device S1. In step S1804, the camera coordinates of the reference point acquired in step S1802 are compared with the camera coordinates of the reference point calculated in step S1803, and if the difference between the two is smaller than a predetermined threshold, the processing ends. If the difference between the two is equal to or greater than the predetermined threshold, a warning indicating that the camera position or attitude has shifted is displayed on the display 156, for example, in step S1805.

[0058] The processing in Fig. 18 may be performed repeatedly at all times while the hydraulic excavator 100 is in operation, or may be performed when the marker 1701 is photographed by the image capturing device S1 for the first time after the hydraulic excavator 100 is started. Alternatively, the processing may be performed as processing to be started when a deviation in the position or attitude of the image capturing device S1 is suspected. In this case, if the deviation of the reference point in step S1804 is smaller than a threshold value, it may be displayed on the display 156 that no deviation in the position or attitude of the image capturing device S1 has occurred. If the operator confirms a deviation in the position or attitude of the image capturing device S1, the operator may perform the calibration processing again to obtain an appropriate position and attitude of the image capturing device S1, or the processing in Fig. 18 may be performed after adjusting the position and attitude of the image capturing device S1.

[0059] 18, the determination in S1804 is made using the deviation of the reference points on the camera coordinates, but instead of this, for example, information on six or more reference points may be acquired, and the position and orientation of the image capture device S1 may be calculated again, and S1804 may be made using the deviation from the originally stored position and orientation. In this case, when displaying the warning in step S1805, it may be possible to display which parameters of the position and orientation of the image capture device S1 have deviated and by how much.

[0060] FIG. 19 shows an example of detecting a deviation in the position and attitude of the image capturing device S1 using a trajectory of reference points. In FIG. 18 , a marker 1701 is used to detect a deviation in the position and attitude of the image capturing device S1 from information about the reference point at one swing angle. In contrast, in the example of FIG. 19 , a trajectory 1901 of reference points is acquired by acquiring the camera coordinates of the marker 1701 while performing a swing operation. For example, this trajectory information may be stored in advance in the hard disk drive 154 before shipping the hydraulic excavator 100, and deviations in the position and attitude of the image capturing device S1 may be determined from deviations in the trajectory. Because the trajectory 1901 includes feature quantities such as curvature and endpoints, these may be used to determine deviations. Furthermore, the information about the trajectory 1901 may be used to calibrate the position and attitude of the image capturing device S1. This process enables calibration of the image capturing device S1 without acquiring attitude information about the hydraulic excavator 100.

[0061] <Regarding Modifications of the Present Disclosure> The present disclosure is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present disclosure, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. Furthermore, part or all of the above-described configurations, functions, processing units, processing means, etc. may be realized in hardware, for example, by designing them as integrated circuits.

[0062] In the above embodiments, the control device 103 can be configured by hardware such as a circuit device that implements its functions, or can be configured by a computing device such as a CPU (Central Processing Unit) that executes software that implements its functions. The control device 103 may be configured as a remote computer that is configured to communicate with the hydraulic excavator 100, or may be mounted on the hydraulic excavator 100 as described in the above embodiments. In either case, the control device 103 can operate as a calibration device that calibrates the imaging device S1.

[0063] In the above embodiment, the hydraulic excavator 100 has been described as a so-called backhoe, but a configuration similar to that of the above embodiment can also be adopted in the case of a loading shovel in which the bucket faces forward.

[0064] In the above embodiment, the reference point coordinate acquisition unit 1031 may automatically identify a feature point other than the marker 1701. For example, by searching for a part having a predetermined feature in the image of the hydraulic excavator 100, the part having the feature can be used as a reference point.

[0065] DESCRIPTION OF SYMBOLS 100... Hydraulic excavator 101... Traveling body 102... Swinging body 103... Control device 111... Boom 112... Arm 113... Bucket 200... Vessel 1031... Reference point coordinate acquisition unit 1032... Reference point coordinate calculation unit 1033... Calibration unit 1034... Support information generation unit 1035... Drive control unit 1701... Marker S1... Imaging device S2... Surrounding sensor S3... Attitude sensor D1... Swing drive unit D2... Front drive unit

Claims

1. A calibration method for calibrating the mounting position and attitude of an imaging device attached to a rotating body of a construction machine having a running body and a rotating body mounted on the running body, comprising the steps of: acquiring an image including at least a part of the construction machine using the imaging device; acquiring the coordinates of a reference point in the image using a predetermined location on the construction machine as the reference point; rotating the rotating body relative to the running body; and calibrating the mounting position and attitude of the imaging device using information about the reference point in the image acquired by the imaging device when the rotating body is rotated.

2. The calibration method according to claim 1, characterized in that in the step of acquiring the coordinates of the reference points, the coordinates of a plurality of the reference points are acquired, and at least one of the acquired plurality of reference points is a point on the surface of the vehicle.

3. A calibration method as described in claim 1, characterized in that in the step of acquiring the coordinates of the reference points, the coordinates of a plurality of the reference points are acquired, at least one of the acquired plurality of reference points is a point on the surface of the traveling body, and at least one of the acquired plurality of reference points is a point on the surface of the rotating body.

4. A calibration method as described in claim 1, characterized in that the imaging device is configured to acquire optical images, and in the step of rotating the rotating body relative to the running body, the rotating body is rotated so that the angular difference between the optical axis of the imaging device and the normal to the surface of the construction machine on which the reference point is located is 45° or less.

5. A calibration method as described in claim 1, characterized in that the imaging device is attached to the side of the rotating body or behind the center of the rotating body in the fore-and-aft direction, and the maximum proportion of the front part of the construction machine that can be included in the image is 20% or less of the image.

6. A calibration method according to claim 1, characterized in that in the step of calibrating the imaging device, the rotation angle is obtained from an attitude sensor that measures the rotation angle of the rotating body, and the mounting position and attitude of the imaging device are calibrated using the obtained rotation angle and the reference point, and in the step of rotating the rotating body relative to the running body, the attitude sensor is initialized each time the rotation angle reaches a predetermined angle, and the rotation angle and coordinates of the reference point are obtained each time the rotation angle reaches the predetermined angle.

7. A calibration method according to claim 1, characterized in that in the step of acquiring the coordinates of the reference point, the coordinates in the image of a point selected by a user in the image displayed on a display device are acquired as the coordinates of the reference point.

8. A calibration method according to claim 1, characterized in that it comprises the steps of: estimating the attitude of the running body based on the attitude of the rotating body; generating support information to assist in operating the construction machine based on the mounting position and attitude of the imaging device calibrated in the calibration step and the estimated attitude of the running body; and superimposing the support information on an image acquired by the imaging device and displaying it on a display device.

9. A calibration method as described in claim 1, characterized in that in the step of acquiring the coordinates of the reference point, a location on the surface shape of the construction machine that has a predetermined characteristic is used as the reference point, and the location with the predetermined characteristic is at least one of a marker attached to the construction machine or a part provided on the construction machine.

10. A calibration method according to claim 9, characterized in that it comprises, after the calibration step, a step of detecting the coordinates of the reference point in the image by image analysis processing of the image, a step of estimating the coordinates of the reference point in the image from the posture of the construction machine, a step of determining whether or not a deviation between the coordinates of the reference point detected by the image analysis processing and the estimated coordinates of the reference point is equal to or greater than a threshold, and a step of outputting a warning if the deviation is equal to or greater than the threshold.

11. A calibration method as described in claim 9, characterized in that the steps of acquiring the image, acquiring the coordinates of the reference points, and rotating the rotating body relative to the running body are repeatedly performed without the user operating the construction machine until coordinates in the image are acquired for a predetermined number of or more of the reference points.

12. A calibration method according to claim 9, characterized in that it comprises, after the calibration step, a step of detecting the coordinates of the reference point in the image by image analysis processing of the image, a step of recalibrating the imaging device using the coordinates of the reference point detected by the image analysis processing, a step of determining whether or not a difference between the mounting position and orientation of the imaging device calibrated by the calibrating step and the mounting position and orientation of the imaging device recalibrated by the recalibrating step is equal to or greater than a threshold, and a step of outputting a warning if the difference is equal to or greater than the threshold.

13. A calibration method according to claim 9, comprising the steps of: reading out the results of recording the trajectory of the reference point in the image while rotating the rotating body; re-recording the trajectory after recording it while rotating the rotating body; and determining whether the mounting position or attitude of the imaging device has shifted since recording by comparing the recording with the re-recording.

14. A calibration device for calibrating the mounting position and attitude of an imaging device attached to a rotating body of a construction machine having the rotating body mounted on a running body, comprising: a reference point coordinate acquisition unit that acquires the coordinates of a predetermined location on the construction machine as a reference point in an image including at least a part of the construction machine acquired by the imaging device; and a calibration unit that calibrates the imaging device using information about the reference point, wherein the calibration unit calibrates the mounting position and attitude of the imaging device using information about the reference point in an image acquired by the imaging device when the rotating body is rotated.

15. A construction machine having a running body and a rotating body mounted on the running body, comprising: an imaging device attached to the rotating body; a reference point coordinate acquisition unit that acquires the coordinates of a predetermined location on the construction machine as a reference point in an image including at least a part of the construction machine acquired by the imaging device; and a calibration unit that calibrates the imaging device using information about the reference point, wherein the calibration unit calibrates the mounting position and attitude of the imaging device using information about the reference point in an image acquired by the imaging device when the rotating body is rotated.

Images