Calibration method, calibration device, and construction machine

Abstract

To provide a method and a system for calibrating a position and an attitude of an imaging device even when a range in which an image of a bucket can be acquired is limited in the imaging device installed in a construction machine.SOLUTION: A calibration method according to the present disclosure includes, in a construction machine in which a swing body is mounted on a traveling body, acquiring an image by an imaging device attached to the swing body, acquiring coordinates of a reference point in the image with a predetermined location of the construction machine as the reference point, swinging the swing body with respect to the traveling body, and calibrating the imaging device using information of the reference point.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to a technique for calibrating an imaging device provided in a construction machine. [Background technology]

[0002] A method for determining the position and orientation of a bucket of a construction machine (excavator) is described, for example, in Patent Document 1, in which an image capturing device (camera) mounted on the vehicle body acquires a bucket image including a preset marker mounted on the bucket, thereby determining the position and orientation information of the image capturing device relative to the bucket, and converting the position and orientation information of the camera relative to the bucket into position and orientation information of the bucket relative to the vehicle body. This makes it possible to calibrate the position and orientation of the image capturing device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-140696 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned Patent Document 1, the imaging device captures an image of the bucket, thereby determining information about the position and orientation of the imaging device. However, depending on the installation position of the imaging device, the range in which the image of the bucket can be captured may be limited, or it may not be possible to capture an image of the bucket at all. In such cases, it is not possible to determine information about the position and orientation of the imaging device.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide a method and device for calibrating the position and attitude of an imaging device installed on a construction machine, even when the range in which bucket images can be acquired is limited. [Means for solving the problem]

[0006] The calibration method disclosed herein involves a construction machine having a rotating body mounted on a running body, acquiring an image using an imaging device attached to the rotating body, obtaining the coordinates of a reference point in the image using a predetermined location on the construction machine as a reference point, rotating the rotating body relative to the running body, and calibrating the imaging device using the information on the reference point. [Effects of the Invention]

[0007] According to the calibration method of the present disclosure, the position and attitude of the imaging device can be calibrated even when the imaging device installed on a construction machine has a limited range in which it can acquire images of the bucket. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating a configuration of a hydraulic excavator 100 according to a first embodiment. [Figure 2] FIG. 2 is a top view of the rotating body 102. [Figure 3] 1 shows an example of the configuration of the control device 103. [Figure 4] 10A and 10B are diagrams illustrating a method for estimating the position and orientation of an image capture device S1. [Figure 5] An example of a location where the imaging device S1 is attached to the hydraulic excavator 100 is shown. [Figure 6] An example of a location where the imaging device S1 is attached to the hydraulic excavator 100 is shown. [Figure 7] This is an example in which an image captured by the imaging device S1 is displayed on the display 156 when the imaging device S1 is attached as described with reference to FIGS. [Figure 8] 10 is a flowchart illustrating a procedure for calibrating the position and orientation of the image capture device S1. [Figure 9] 10 is an example of a table for storing data required to configure the position and orientation of the image capture device S1. [Figure 10] 10 is an example of a data table for storing the position and orientation of the image capture device S1. [Figure 11] FIG. 10 is a diagram showing a preferred example of a reference point. [Figure 12] This is an example of a combination of a reference point placed on a traveling body and a reference point placed on a rotating body. [Figure 13] 10 is a flowchart illustrating a procedure for calibrating the imaging device S1 by combining reference points placed on the traveling body and reference points placed on the rotating body. [Figure 14] An example of acquiring a reference point when a preset turning angle is reached is shown below. [Figure 15] This is an example in which support information is superimposed on an image captured by the imaging device S1 and displayed on the display 156. [Figure 16] 10 is a flowchart illustrating a procedure for superimposing support information on a video captured by the imaging device S1 and displaying the superimposed information on the display 156. [Figure 17] In this example, a marker 1701 is placed on the side of a vehicle 101 in order to automatically detect a reference point. [Figure 18] 10 is an example of a flowchart illustrating a procedure for automatically detecting a shift in the position or orientation of the image capturing device S1 by automatically detecting a shift in the position of a reference point. [Figure 19] An example will be shown in which deviations in the position and orientation of the image capture device S1 are detected using the locus of reference points. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be designated by the same numerals. Note that the accompanying drawings show embodiments and implementation examples according to the principles of the present disclosure, but these are for understanding the present disclosure and are not to be used to interpret the present disclosure in a limiting manner. The descriptions in this specification are merely typical examples and are not intended to limit the scope or application of the present disclosure in any way.

[0010] In the following embodiments, the description is given in sufficient detail to enable a person skilled in the art to implement the present disclosure, but it should be understood that other implementations and forms are possible, and that changes in configuration and structure and substitutions of various elements are possible without departing from the scope and spirit of the technical idea of ​​the present disclosure. Therefore, the following description should not be interpreted as being limited thereto.

[0011] <First Embodiment> FIG. 1 is a diagram illustrating the configuration of a hydraulic excavator 100 according to a first embodiment of the present disclosure as one type of construction machine. The hydraulic excavator 100 is, as an example, a construction machine having a backhoe-type front structure in which a bucket 113 faces an operator cabin OR. Operation of the hydraulic excavator 100 excavates ground L, which is an excavation target, such as a plane L0, a plane L1, or a slope L2. The soil excavated by the hydraulic excavator 100 is discharged (released) into a vessel 200.

[0012] As an example, the hydraulic excavator 100 has a lower traveling body (hereinafter simply referred to as the traveling body) 101, and an upper rotating body (hereinafter simply referred to as the rotating body) 102 that is rotatably supported on the traveling body 101. The rotating body 102 is also provided with a cab OR that forms an operation room where an operator sits, a control device 103, and various sensors (an imaging device S1, a surrounding sensor S2, and an attitude sensor S3).

[0013] The hydraulic excavator 100 also has a front working mechanism (hereinafter simply referred to as the working mechanism) 130 consisting of a boom 111 attached to the revolving body 102, an arm 112 attached to the tip of the boom 111, and a bucket 113 attached to the tip of the arm 112. The hydraulic excavator 100 performs excavation work on the ground surface L by driving the working mechanism 130 (boom 111, arm 112, bucket 113) by an engine (power source), hydraulic cylinder (actuator), etc. (not shown).

[0014] The running body 101 has a frame and a track wound around the frame, and is a component that causes the hydraulic excavator 100 to travel when the track is rotated by a travel motor. The rotating body 102 performs an operation of rotating 360 degrees (rotating operation) relative to the running body 101. The direction, speed, and angle of the rotating operation are controlled by, for example, an operator.

[0015] A control device 103 is mounted on the revolving unit 102. The control device 103 acquires images using an imaging device S1 including an imaging element, and generates support information including the current positions of the bucket 113, the traveling body 101, the work ground surface, and the vessel 200 such as a dump truck, based on sensor information obtained from a surrounding sensor S2 and an attitude sensor S3.

[0016] FIG. 2 is a top view of the revolving unit 102. An imaging device S1, a surrounding sensor S2, and an attitude sensor S3 are arranged on the revolving unit 102 to detect the situation around the hydraulic excavator 100 and the attitude of the hydraulic excavator 100. As shown in FIG. 2, as an example, the imaging device S1 is attached to the left side surface of the revolving unit 102 of the hydraulic excavator 100 at a position protruding from the left side surface of the building of the revolving unit 102. The imaging device S1 has an imaging surface (receiving surface or light-receiving surface) such as a CMOS sensor oriented to capture an image of a field of view SR1 looking forward (i.e., forward left) from the left side surface of the revolving unit 102. In this embodiment, the imaging device S1 is configured to acquire an image including an area including at least a portion of the traveling unit 101 and the planned traveling path of the traveling unit 101.

[0017] The surrounding sensor S2 is an object detection sensor such as a LiDAR (Light Detection and Ranging) or ultrasonic sensor that detects the presence or absence of an object and its distance within the field of view SR2. The surrounding sensor S2 is not limited to this and may be composed of multiple LiDAR or ultrasonic sensors. Furthermore, the imaging device S1 and the surrounding sensor S2 may be composed of a single or multiple stereo cameras capable of imaging and stereoscopic vision. The attitude sensor S3 is a sensor that detects the attitude of the hydraulic excavator 100 and is capable of acquiring the attitudes of the boom 111, arm 112, and bucket 113 and the rotation angle of the rotating unit 102 relative to the traveling unit 101. The attitude sensor S3 may be composed of, for example, a gyro sensor, an inclination sensor, an acceleration sensor, or a combination of these. Furthermore, the example illustrated in FIG. 2 is just one example, and the present disclosure is not limited thereto.

[0018] 2, in addition to the imaging device S1 that captures an image of the left front field of view SR1 described above, an imaging device S11 that captures an image of the right front field of view SR11 and an imaging device S12 that captures an image of the front field of view SR12 are attached to the rotating unit 102. For example, the hydraulic excavator 100 includes multiple cameras corresponding to the imaging devices S1, S11, and S12.

[0019] For example, in this embodiment, the imaging device S11 is attached to the right side of the revolving unit 102 at a position protruding from the right side of the building of the revolving unit 102, and is configured to capture an image of an area to the right front of the running unit 101, including at least a portion of the running unit 101. In addition, the imaging device S12 is attached to the front of the revolving unit 102 (the surface to which the work implement 130 is attached; in this embodiment, the lower part of the cab OR) at a position protruding from the front of the building of the revolving unit 102, and is configured to capture an image of an area to the front front of the running unit 101, including at least a portion of the running unit 101 and at least a portion of the work implement 130.

[0020] The imaging devices S1, S11, and S12 are not limited to having the above-described configuration. For example, the imaging areas of the imaging devices S1, S11, and S12 are not limited to the area viewed in three directions from the running body 101. For example, imaging devices that capture images of the entire surroundings may be provided by adjusting the installation position, angle, and number of imaging devices. For example, in addition to the imaging devices S1, S11, and S12, an imaging device configured and arranged to capture an area behind the running body 101, including at least a portion of the running body 101, may be provided on the rear surface of the building of the rotating body 102 or at a position toward the rear of the center of the bottom surface. By adding this rear imaging device, it is possible to capture images of both the areas in front and behind the running body 101. This makes it possible for the imaging device to capture an image including a portion of the running body 101 and the area along its travel path, regardless of whether the running body 101 is moving forward or backward.

[0021] In other words, in this embodiment, each of these imaging devices needs only to be positioned on the rotating body 102 in a position and attitude such that it captures an image of an area including at least a portion of the running body 101 and the planned running path of the running body 101.

[0022] 3 shows an example configuration of the control device 103. As an example, the control device 103 includes a reference point coordinate acquisition unit 1031, a reference point coordinate calculation unit 1032, a calibration unit 1033, a support information generation unit 1034, and a drive control unit 1035. The control device 103 includes, for example, a CPU 151 (processor), a ROM 152, a RAM 153, and a hard disk drive 154. An input device 155 and a display 156 (display unit) are provided in the cab OR of the revolving unit 102. The control device 103 performs information processing based on information from the input device 155 and various sensors, outputs the results to the display 156, and controls the operation of the hydraulic excavator 100. The CPU 151 is one aspect of a processor, and may be a GPU or other semiconductor device capable of arithmetic processing, or a combination thereof.

[0023] The reference point coordinate acquisition unit 1031 acquires the coordinates of the reference point required to calibrate the position and attitude of the imaging device S1 in the image acquired from the imaging device S1. To acquire the coordinates of the reference point, as an example, the attitude of the construction machine 100 is controlled so that the imaging device S1 includes the tip of the traveling body 101, and the user selects the tip of the crawler on the image displayed on the display 156 with the cursor. This makes it possible to acquire the coordinates of the reference point in the image.

[0024] The reference point coordinate calculation unit 1032 calculates the position of the tip of the crawler in three-dimensional space based on the three-dimensional information acquired from the surrounding sensor S2 and the attitude information acquired from the attitude sensor S3.

[0025] The calibration unit 1033 estimates the position and attitude of the imaging device S1 using the coordinates on the display of the tip of the crawler and the information on its position in three-dimensional space acquired above. At this time, since multiple pieces of information that serve as reference points are required as will be described later, the drive control unit 1035 drives the rotation drive unit D1 to change the rotation angle of the rotating body relative to the running body, change the position of the reference point on the display, and acquire the information on the reference point again.

[0026] The support information generation unit 1034 generates support information to be superimposed on the image captured by the imaging device S1 from the information on the position and attitude of the imaging device S1 estimated by the calibration unit 1033 and the information on the attitude of the hydraulic excavator 100 acquired by the attitude sensor S3, and outputs this support information to the display 156 while superimposing it on the image. For example, when the traveling object 101 is captured as an image by the imaging device S1, the support information generation unit 1034 generates guide lines indicating the traveling direction of the traveling object 101 as support information using the turning angle information acquired by the attitude sensor S3. By outputting this support information to the display 156 while superimposed on the image captured by the imaging device S1, the occupant can easily confirm the traveling direction of the hydraulic excavator 100.

[0027] FIG. 4 is a diagram illustrating a method for estimating the position and orientation of the image capture device S1. Here, the image capture device S1 is a general camera, and the internal parameter K of the camera is calculated based on a known focal length ( f x , f y ) and the principal point position ( c x , c y ) is used to express it as Equation 1. It is assumed that the distortion of the camera lens has been corrected, and the sample points in the world coordinate system are known using the three-dimensional information and posture information of the vehicle model of the hydraulic excavator 100.

[0028]

number

[0029] The external parameters of the camera, which are the information we want to obtain, are extrinsic Assuming that the orientation R and the position t → The combination of T extrinsic The transformation matrix from world coordinates to camera coordinates is T projection This results in the relationship shown in Equation 2.

[0030]

number

[0031] The sample points in the world coordinate system are r → , the transformation matrix from world coordinates to camera coordinates is T projection Then, the sample point p within the field of view captured by the camera → is expressed by Equation 3. λ is a coefficient used when converting from world coordinates to camera coordinates.

[0032]

number

[0033] From Equation 3, three equations are generated for each sample point, but the unknown constant λ increases by one. projectionis expressed as a combination of a 3x3 matrix that represents the orientation and a 3x1 matrix that represents the position, so there are a total of 12 unknown parameters. However, since the equation is equivalent even if multiplied by a constant, the number of unknowns becomes 11. Therefore, if n samples are taken, the relationship in Equation 4 holds, and if six or more samples are taken, T projection is obtained, and it can be seen that the camera's orientation and position can be estimated using Equation 2. The above is generally called the PnP (Perspective n Point) problem.

[0034]

number

[0035] 5 and 6 show an example of where the imaging device S1 is attached to the hydraulic excavator 100. Fig. 5 shows the hydraulic excavator 100 as seen from the side, and Fig. 6 shows the hydraulic excavator 100 as seen from above. In this example, in order to easily check the direction in which the running body 101 is traveling while the hydraulic excavator 100 is traveling, the imaging device S1 is attached at a position lower than the operation cabin OR of the rotating body 102 so as to provide a field of view in front of the rotating body, and is also attached outward of the running body 101 as seen from above. This makes it possible to check the tip of the running body 101 when the rotating body is facing forward or backward relative to the running body.

[0036] As will be described later, support information can be superimposed on the image captured by the imaging device S1 and displayed on the display 156. If information about the position and attitude of the imaging device S1 is not properly retained at this time, the state of the hydraulic excavator 100 displayed on the display 156 cannot be properly calculated, resulting in a discrepancy in the display of the support information. Therefore, a calibration method is required to properly obtain information about the position and attitude of the imaging device S1 when the imaging device S1 is attached to the hydraulic excavator 100. Even after calibration, a discrepancy in the position or attitude of the imaging device S1 may occur due to unintentional contact or the like. In such cases, it is desirable to have a means for recalibration or for automatically detecting the discrepancy. A message prompting calibration may be displayed on the display 156 when the system is started for the first time after the imaging device S1 is attached to the hydraulic excavator 100, or a message prompting calibration may be displayed again on the display 156 when a discrepancy in the position or attitude of the imaging device S1 is detected. Furthermore, when a deviation in the position or orientation of the image capture device S1 is detected, calibration may be performed automatically as described below, and newly obtained information on the position and orientation of the image capture device S1 may be used to superimpose support information and display it on the display 156. Furthermore, an operation mode for performing calibration may be provided, and the user may actively perform calibration.

[0037] 7 shows an example of an image captured by the imaging device S1 displayed on the display 156 when the imaging device S1 is installed as described in FIGS. 5 and 6. The front portion of the boom 111, arm 112, and bucket 113 are hidden by the rotating unit 102 and cannot be seen from the imaging device S1. Therefore, it is not possible to calibrate the position and attitude of the imaging device S1 using, for example, the tip of the bucket 113 as a reference point. For example, if the front portion is only visible within 20% of the angle of view of the imaging device S1, it is difficult to calibrate the imaging device using a part of the front portion as a reference point. Therefore, calibration is performed by using, for example, the tip of the traveling unit 101 as a reference point 701.

[0038] FIG. 8 is a flowchart illustrating a procedure for calibrating the position and orientation of the image capturing device S1. In step S801, the image capturing device S1 captures an image and outputs the image on the display 156. In step S802, the user uses the input device 155 to operate the cursor on the display 156 and click a reference point on the image output on the display 156. The reference point coordinate acquisition unit 1031 acquires the clicked coordinate as the coordinate of the reference point in the image. In step S803, the orientation sensor S3 acquires the orientation of the shovel and calculates the world coordinate corresponding to the selected reference point. In step S804, the reference point coordinate acquisition unit 1031 determines whether the required number of reference points have been acquired. As mentioned above, information on at least six reference points is required. However, since the greater the number of reference points, the higher the calibration accuracy, a greater number of reference points may be set as the threshold. If the required number of reference points has not been acquired in step S804, the drive control unit 1035 rotates the rotating body relative to the running body via the rotation drive unit D1 in step S805, and then repeats steps S801 to S804. The rotation in step S805 only needs to be performed within a range in which the reference points are expected to be within the display; for this determination, initial values ​​may be assigned to the position and attitude of the camera, and the rotating body may be rotated within the expected range based on these initial values. If the required number of reference points has been acquired in step S804, the calibration unit 1033 calibrates the image capture device S1 in step S806.

[0039] 9 shows an example of a table that stores data necessary to determine the position and orientation of the image capture device S1. This data table can be stored in the hard disk drive 154. This data table holds the camera coordinates acquired in step S802 and the world coordinates calculated in step S803, each for the number n of necessary reference points. Based on this, the calibration unit 1033 calibrates the image capture device S1.

[0040] 10 shows an example of a data table for storing the position and attitude of the image capture device S1. This data table can be stored in the hard disk drive 154. This data table stores a total of six parameters: X, Y, and Z for position, and θp, θy, and θr for attitude, which correspond to pitch angle, yaw angle, and roll angle. The data table may be pre-stored with initial values ​​corresponding to the mounting position and attitude of the image capture device S1, or estimated results of the position and attitude obtained as a result of calibration may be added to or overwritten.

[0041] FIG. 11 is a diagram showing a preferred example of a reference point. When the tip of FIG. 7 is the reference point, the angle between the normal to the plane on which the reference point is located (in this example, the side of the vehicle 101) and the optical axis of the image capture device S1 is nearly a right angle, making it difficult to accurately select the reference point on the image on the display 156. In this case, the accuracy of the position and orientation of the image capture device S1 obtained by calibration may also be poor. In FIG. 11, the image capture device S1 attached to the left side of the rotating unit captures an image of the tip of the vehicle 101 on the right side of the running unit. In this case, the angle between the normal to the side of the vehicle 101, including the reference point, and the optical axis of the image capture device S1 is nearly parallel, making it possible to accurately select the reference point on the image on the display 156. Therefore, the accuracy of the position and orientation of the image capture device S1 obtained by calibration is also good.

[0042] As an example, calibration accuracy can be improved by acquiring a reference point at a rotation angle such that the angular difference between the normal to the plane containing the reference point and the optical axis of the image capture device S1 is 45° or less. Since six or more reference points must be acquired, the angular difference between the normal to at least one of the reference points and the optical axis of the image capture device S1 may be 45° or less, or all reference points may satisfy the same condition. Furthermore, it is not necessary to acquire six or more reference points using only reference point 1101 (i.e., only reference points on the vehicle 101). Reference points may be acquired by combining reference point 701 and reference point 1101, or a reference point may be placed at the rear end of the crawler. Furthermore, the reference point does not necessarily have to be at the tip of the vehicle 101; it may be anywhere on the vehicle that can be photographed by the image capture device S1.

[0043] 12 shows an example in which a reference point placed on a traveling body and a reference point placed on a rotating body are combined. Here, a case is shown in which part of the bucket 113 is visible to the imaging device S1 depending on the posture of the boom 111, arm 112, and bucket 113, and one of the toes of the bucket 113 is set as the reference point 1201. Also, the reference point does not necessarily have to be placed on the bucket 113; for example, a distinctive part such as a corner of the rotating body may be set as the reference point 1202. By combining these reference points placed on the rotating body and the reference points placed on the traveling body and acquiring information on six or more reference points, the imaging device S1 can be calibrated.

[0044] 13 is a flowchart illustrating the procedure for calibrating the image capture device S1 by combining a reference point placed on the traveling body and a reference point placed on the rotating body. S801, S802, S803, S805, and S806 are the same as those described in FIG. 8, and therefore will not be described again.

[0045] In step S1301, the reference point coordinate acquisition unit 1031 determines whether the number of reference points to be acquired on the predetermined traveling body has been acquired. If the required number of reference points on the traveling body has not been acquired, the process of acquiring the required reference points on the traveling body is repeated by changing the rotation angle in step S805. If the required number of reference points on the traveling body have been acquired, in step S1302 the reference point coordinate acquisition unit 1031 determines whether the required number of reference points have been acquired on the rotating body. If the required number of reference points on the rotating body have not been acquired, in step S1303 the drive control unit 1035 changes the position of the bucket via the front drive unit D2, and then repeats the processes from S801 to S1302. If the required number of reference points have been acquired on the rotating body, in step S806 the calibration unit 1033 calibrates the image capture device S1.

[0046] FIG. 14 shows an example in which a reference point is acquired when a preset rotation angle is reached. The rotation angle can be acquired by using a rotation angle sensor as the attitude sensor S3. However, depending on the sensor's specifications and configuration, errors may occur during rotation, and the sensor may be forcibly reset when a predetermined rotation angle is reached. If rotation angle information is acquired while an error is present, the reference point coordinate calculation unit 1032 calculates the position of the reference point under that error, degrading the accuracy of the calibration of the position and attitude of the image capture device S1 by the calibration unit 1033. Therefore, by pre-storing a rotation angle at which the rotation angle is forcibly reset and acquiring attitude information at such a rotation angle, it is possible to prevent accuracy degradation. Here, n rotation angles at which the sensor is reset are pre-set, and each rotation angle θi is distinguished by a counter i.

[0047] In step S1401, i is set to an initial value of 1. In step S1402, the drive control unit 1035 drives the rotation drive unit D1 so that the i-th rotation angle θi is obtained. Steps S801 to S804 are the same as those described in FIG. 8, and therefore their description will be omitted. If the required number of reference points has not been obtained in step S804, the counter i is incremented in step S1403, and the processing of steps S1402 to S804 is repeated. If the required number of reference points has been obtained in step S804, the calibration unit 1033 calibrates the imaging device S1 in step S806.

[0048] In Figure 14, resetting the sensor refers to the process of overwriting the turning angle detected by the sensor with a predetermined initial value. The initial value may be a different value for each turning angle at which the sensor is reset, or may be the same value for all turning angles. The process of resetting the sensor may be performed by the sensor itself, for example, or by any other computing device.

[0049] FIG. 15 shows an example in which support information is superimposed on an image captured by the imaging device S1 and displayed on the display 156. The traveling direction of the traveling body 101 is represented as support information 1501 by a series of arrow feathers with the tip pointing in the traveling direction of the hydraulic excavator 100, which are arranged along the edge of the surface of the traveling body 101 that is in contact with the ground. The support information 1501 allows the operator to easily confirm the traveling direction of the hydraulic excavator 100, thereby improving safety and reducing the burden on the operator. The operator may operate the hydraulic excavator 100 from inside the hydraulic excavator 100, or may transmit information about the hydraulic excavator 100 to a remote control room and operate the hydraulic excavator 100 while checking the display in the remote control room. In particular, when remotely operating the hydraulic excavator 100, information about the environment surrounding the hydraulic excavator 100 is limited, so the burden on the operator can be significantly reduced by superimposing support information.

[0050] A further explanation of the support information 1501 is provided. If the correct position and orientation of the imaging device S1 are known, and the orientation of the excavator body is also known, the direction in which the traveling body 101 should be facing can also be determined. Information instructing the hydraulic excavator 100 to proceed in that direction can be presented as support information 1501. More specifically, if the position and orientation of the imaging device S1 are known, it is possible to calculate Tprojection, which can be used to convert the three-dimensional coordinates of the support information into two-dimensional coordinates on the display 156. In other words, accurate support information 1501 can be presented based on the calibration results of the imaging device S1.

[0051] 16 is a flowchart illustrating the procedure for superimposing support information on video captured by the imaging device S1 and displaying it on the display 156. In step S1601, the imaging device S1 acquires video. In step S1602, the attitude sensor S3 acquires the attitude of the shovel. In step S1603, the support information generation unit 1034 generates support information from the information about the attitude of the shovel acquired in step S1602 and the information about the position and attitude of the camera estimated by the calibration unit 1033. As described above, one example of the support information generated here is the support information 1501 shown in FIG. 15. In step S1604, the support information generated in step S1603 is superimposed on the video captured in step S1601 and displayed on the display 156.

[0052] <First embodiment: Summary> The hydraulic excavator 100 according to the first embodiment acquires an image from an imaging device S1 attached to a rotating body of the hydraulic excavator 100, acquires the coordinates of a predetermined location on the hydraulic excavator 100 in the image as a reference point, rotates the rotating body, and calibrates the position and attitude of the imaging device S1 using the reference point coordinates in the image. By using the reference point on the hydraulic excavator 100 while rotating the rotating body, it is possible to calibrate the position and attitude of the imaging device S1 even if the range in which the imaging device S1 acquires images is limited.

[0053] <Embodiment 2> In the first embodiment, it has been described that the operator visually checks the reference point 701 or the reference point 1101 on the display 156 and selects it using the input device 155. In the second embodiment of the present disclosure, a configuration example will be described in which the control device 103 automatically calculates the coordinates of the reference point on the display 156. The other configurations are the same as those in the first embodiment.

[0054] FIG. 17 shows an example in which a marker 1701 is placed on the side of the running object 101 to automatically detect a reference point. In the first embodiment, the tip of the running object 101 is used as the reference point. However, since a fixed position is preferable when automatically detecting a reference point, the marker 1701 is placed on a non-moving portion of the running object 101. The type of marker 1701 may be a checkered pattern as shown in FIG. 17, or a two-dimensional barcode or the like may be used to distinguish multiple reference points. By using such a distinctive pattern, when the imaging device S1 captures the marker 1701, the control device 103 can automatically calculate the position of the marker 1701 by image processing. Furthermore, such image processing may calculate not only the coordinates of the marker 1701 but also a normal vector of the marker 1701, and use the normal vector to improve the accuracy of calibration.

[0055] As described above, in the processing flow of Fig. 8, the acquisition of the reference point in step S802 can be performed automatically without user operation, thereby shortening the calibration time. Furthermore, by improving the accuracy of automatic detection of the marker 1701, it is also possible to calculate the position and orientation of the image capture device S1 with high accuracy. Furthermore, in the processing flow of Fig. 8, the entire calibration process can be automated by automatically performing the rotation operation in step S805 by the drive control unit 1035. Also, although an example of detecting the reference point using the marker 1701 has been shown here, it is also possible to extract a characteristic location in the appearance of the hydraulic excavator 100 by image processing and use it instead of the reference point without using the marker 1701.

[0056] 18 is an example of a flowchart illustrating a procedure for automatically detecting a shift in the position or orientation of the image capture device S1 by automatically detecting a shift in the position of the reference point. Here, it is assumed that calibration of the image capture device S1 has been completed in advance, and the estimated results of the position and orientation of the image capture device S1 have been stored in the table shown in FIG.

[0057] In step S1801, the imaging device S1 acquires an image including the reference point. In step S1802, the control device 103 performs image processing to acquire the position of the marker 1701 as the camera coordinates of the reference point. In step S1803, the attitude sensor S3 acquires the attitude of the shovel. Furthermore, the reference point coordinate calculation unit 1032 calculates the camera coordinates corresponding to the reference point using information on the calibrated position and attitude of the imaging device S1. In step S1804, the camera coordinates of the reference point acquired in step S1802 are compared with the camera coordinates of the reference point calculated in step S1803, and if the difference between the two is smaller than a predetermined threshold, the processing ends. If the difference between the two is equal to or greater than the predetermined threshold, a warning indicating that the camera position or attitude has shifted is displayed on the display 156, for example, in step S1805.

[0058] 18 may be repeatedly performed at all times while the hydraulic excavator 100 is in operation, or may be performed when the marker 1701 is photographed by the imaging device S1 for the first time after the hydraulic excavator 100 is started. Alternatively, the process may be performed as a process that is started when a deviation in the position or attitude of the imaging device S1 is suspected. In this case, if the deviation of the reference point in step S1804 is smaller than a threshold value, it may be displayed on the display 156 that no deviation in the position or attitude of the imaging device S1 has occurred. If the operator confirms a deviation in the position or attitude of the imaging device S1, the operator may perform the calibration process again to obtain an appropriate position and attitude of the imaging device S1, or the process of FIG. 18 may be performed after adjusting the position and attitude of the imaging device S1.

[0059] 18, the determination in S1804 is made using the deviation of the reference points on the camera coordinates, but instead of this, for example, information on six or more reference points may be acquired, and the position and orientation of the image capture device S1 may be calculated again, and S1804 may be made using the deviation from the originally stored position and orientation. In this case, when displaying the warning in step S1805, it may be possible to display which parameters of the position and orientation of the image capture device S1 have deviated and by how much.

[0060] FIG. 19 shows an example of detecting a deviation in the position and attitude of the image capturing device S1 using the trajectory of reference points. In FIG. 18, the marker 1701 is used to detect a deviation in the position and attitude of the image capturing device S1 from information about the reference points at one swing angle. In contrast, in the example of FIG. 19, the trajectory 1901 of the reference points is acquired by acquiring the camera coordinates of the marker 1701 while performing a swing operation. For example, this trajectory information may be stored in advance in the hard disk drive 154 before shipping the hydraulic excavator 100, and deviation in the position and attitude of the image capturing device S1 may be determined from deviation in the trajectory. Since the trajectory 1901 includes feature quantities such as curvature and endpoints, these may be used to determine the deviation. Furthermore, the information about the trajectory 1901 may be used to calibrate the position and attitude of the image capturing device S1. This process enables calibration of the image capturing device S1 without acquiring attitude information about the hydraulic excavator 100.

[0061] <Modifications of the present disclosure> The present disclosure is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present disclosure, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits.

[0062] In the above embodiments, the control device 103 can be configured by hardware such as a circuit device that implements its functions, or can be configured by a calculation device such as a CPU (Central Processing Unit) that executes software that implements its functions. The control device 103 may be configured as a remote computer that is configured to communicate with the hydraulic excavator 100, or may be mounted on the hydraulic excavator 100 as described in the above embodiments. In either case, the control device 103 can operate as a calibration device that calibrates the imaging device S1.

[0063] In the above embodiment, the hydraulic excavator 100 has been described as a so-called backhoe, but a configuration similar to that of the above embodiment can also be adopted in the case of a loading shovel in which the bucket faces forward.

[0064] In the above embodiment, the reference point coordinate acquisition unit 1031 may automatically identify feature points other than the marker 1701. For example, by searching for a part having a predetermined feature in the image of the hydraulic excavator 100, the part having the feature can be used as a reference point. [Explanation of symbols]

[0065] 100...hydraulic excavator 101...Traveling body 102...Rotating body 103...Control device 111...Boom 112...Arm 113...Bucket 200…Vessel 1031...Reference point coordinate acquisition unit 1032...Reference point coordinate calculation section 1033...Calibration section 1034…Support information generation unit 1035...Drive control unit 1701...Marker S1: Imaging device S2: Surrounding sensor S3: Posture sensor D1: Swivel drive unit D2...Front drive unit

Claims

1. A calibration method for calibrating the mounting position and attitude of an imaging device attached to a rotating body of a construction machine having a traveling body and a rotating body provided on the traveling body, comprising: acquiring an image including at least a part of the construction machine by the imaging device; a step of acquiring coordinates of a predetermined point of the construction machine within the image as a reference point; rotating the rotating body relative to the traveling body; calibrating the mounting position and attitude of the imaging device using information on the reference point before the rotating body is rotated and information on the reference point after the rotating body is rotated; A calibration method having:

2. In the step of acquiring coordinates of the reference points, coordinates of the plurality of reference points are acquired, At least one of the acquired reference points is a point on the surface of the moving object.

2. The calibration method according to claim 1, wherein the calibration step comprises:

3. In the step of acquiring coordinates of the reference points, coordinates of the plurality of reference points are acquired, At least one of the acquired reference points is a point on a surface of the traveling object, At least one of the acquired reference points is a point on the surface of the rotating body.

2. The calibration method according to claim 1, wherein the calibration step comprises:

4. the imaging device is configured to acquire an optical image; In the step of rotating the rotating body relative to the traveling body, the rotating body is rotated so that the angle difference between the optical axis of the imaging device and the normal to the surface of the construction machine on which the reference point is located is 45° or less.

2. The calibration method according to claim 1, wherein the calibration step comprises:

5. The imaging device is attached to the side of the rotating body or rearward of the center of the rotating body in the fore-and-aft direction, and the maximum proportion of the front part of the construction machine that can be included in the image is 20% or less of the image.

2. The calibration method according to claim 1, wherein the calibration step comprises:

6. In the step of calibrating the imaging device, a rotation angle is acquired from an attitude sensor that measures a rotation angle of the rotating body, and an attachment position and attitude of the imaging device are calibrated using the acquired rotation angle and the reference point; In the step of rotating the rotating body relative to the traveling body, the attitude sensor is initialized each time the rotation angle reaches a predetermined angle, and the rotation angle and the coordinates of the reference point are acquired each time the rotation angle reaches the predetermined angle.

2. The calibration method according to claim 1, wherein the calibration step comprises:

7. In the step of acquiring the coordinates of the reference point, the coordinates in the image of a point selected by a user in the image displayed on a display device are acquired as the coordinates of the reference point.

2. The calibration method according to claim 1, wherein the calibration step comprises:

8. estimating the attitude of the traveling body based on the attitude of the rotating body; a step of generating support information for supporting operation of the construction machine based on the mounting position and attitude of the imaging device calibrated in the calibrating step and the estimated attitude of the traveling body; a step of superimposing the support information on the image acquired by the imaging device and displaying the image on a display device; have 2. The calibration method according to claim 1, wherein the calibration step comprises:

9. In the step of acquiring the coordinates of the reference points, a location having a predetermined characteristic on the surface shape of the construction machine is used as the reference point; The location having the predetermined characteristic is at least one of a marker attached to the construction machine or a part provided on the construction machine.

2. The calibration method according to claim 1, wherein the calibration step comprises:

10. After the calibrating step, detecting the coordinates of the reference point in the image by performing an image analysis process on the image; a step of estimating the coordinates of the reference point in the image from the posture of the construction machine; determining whether or not a deviation between the coordinates of the reference points detected by the image analysis processing and the estimated coordinates of the reference points is equal to or greater than a threshold; outputting a warning if the deviation is equal to or greater than the threshold value; have 10. The calibration method of claim 9.

11. The user does not operate the construction machine until the coordinates in the image are acquired for a predetermined number or more of the reference points. acquiring the image; obtaining the coordinates of the reference point; rotating the rotating body relative to the traveling body; Repeatedly carry out 10. The calibration method of claim 9.

12. After the calibrating step, detecting the coordinates of the reference point in the image by performing an image analysis process on the image; recalibrating the imaging device using the coordinates of the reference points detected by the image analysis process; a step of determining whether a difference between the mounting position and orientation of the imaging device calibrated in the calibrating step and the mounting position and orientation of the imaging device recalibrated in the recalibrating step is equal to or greater than a threshold; outputting a warning when the difference is equal to or greater than the threshold value; have 10. The calibration method of claim 9.

13. a step of reading out a result of recording a trajectory of the reference point in the image while rotating the rotating body; a step of re-recording the trajectory while rotating the rotating body after recording the trajectory; a step of comparing the recording with the re-recording to determine whether the mounting position or the orientation of the imaging device has shifted after the recording; have 10. The calibration method of claim 9.

14. A calibration device for calibrating the mounting position and attitude of an imaging device mounted on a rotating body of a construction machine having the rotating body mounted on a traveling body, a reference point coordinate acquisition unit that acquires the coordinates of a predetermined location of the construction machine as a reference point within an image including at least a portion of the construction machine acquired by the imaging device; a calibration unit that calibrates the imaging device using information about the reference points; Equipped with The calibration unit calibrates the mounting position and attitude of the imaging device using information on the reference point before the rotating body is rotated and information on the reference point after the rotating body is rotated. A calibration device characterized by:

15. A construction machine having a running body and a rotating body provided on the running body, an imaging device attached to the rotating body; a reference point coordinate acquisition unit that acquires the coordinates of a predetermined location of the construction machine as a reference point within an image including at least a portion of the construction machine acquired by the imaging device; a calibration unit that calibrates the imaging device using information about the reference points; Equipped with The calibration unit calibrates the mounting position and attitude of the imaging device using information on the reference point before the rotating body is rotated and information on the reference point after the rotating body is rotated. Construction machinery characterized by:

Images