Camera adjustment system for construction machine, construction machine, camera adjustment method for construction machine, and program

Abstract

To provide a camera adjustment system for a construction machine capable of performing the calibration of a plurality of cameras with high accuracy.SOLUTION: According to one aspect of the present invention, the camera adjustment system is provided for a construction machine. The camera adjustment system includes a processor. The processor is configured to execute a tilt information acquisition step, a correction value calculation step, and a camera correction step. In the tilt information acquisition step, tilt information is acquired from a tilt sensor provided in the construction machine. In the correction value calculation step, a correction value for the tilt of the construction machine is calculated based on the tilt information. In the camera correction step, the orientation of the first camera or the second camera is corrected while referring to the correction value so that the overlap amount between the first correction image acquired from the first camera and the second correction image acquired from the second camera is equal to or greater than a predetermined value. The first camera and the second camera are installed on the main body of the construction machine and have overlapping photographing ranges.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a camera adjustment system for construction machinery, construction machinery, a camera adjustment method for construction machinery, and a program.

Background Art

[0002] A plurality of cameras are installed in construction machinery to grasp the surrounding situation. When a new camera is installed or replaced, it is necessary to calibrate whether each of the plurality of cameras is facing the correct direction. On the other hand, as disclosed in the following documents, a technique for performing correction to form a composite image when synthesizing images of a plurality of cameras is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The above prior art performs correction at the time of image synthesis, and does not correct the direction of the camera. Therefore, there is a limit to the accuracy of calibration.

[0005] In view of the above circumstances, the present invention aims to provide a camera adjustment system for construction machinery that can perform calibration of a plurality of cameras with high accuracy.

Means for Solving the Problems

[0006] According to one aspect of the present invention, a camera adjustment system for a construction machine is provided. This camera adjustment system includes a processor. The processor is configured to execute an inclination information acquisition step, a correction value calculation step, and a camera correction step. In the inclination information acquisition step, inclination information is acquired from an inclination sensor provided in the construction machine. In the correction value calculation step, a correction value for the inclination of the construction machine is calculated based on the inclination information. In the camera correction step, while referring to the correction value, the orientation of the first camera or the second camera is corrected so that the overlap amount between the first correction image acquired from the first camera and the second correction image acquired from the second camera becomes a certain value or more. The first camera and the second camera are installed on the main body of the construction machine, and their shooting ranges overlap each other.

[0007] According to such an aspect, the orientation of the first camera or the second camera can be automatically corrected using the overlap amount between the first correction image and the second correction image. Further, when correcting the orientation of the camera, the inclination of the construction machine is taken into account, so that calibration of a plurality of cameras can be performed with high accuracy.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various features shown in the embodiments described below can be combined with each other.

[0010] First, the program for realizing the software appearing in this embodiment may be provided as a non-transitory computer-readable medium readable by a computer, may be provided so as to be downloadable from an external server, or may be provided so that the program is started on an external computer and its function is realized on a client terminal (so-called cloud computing).

[0011] In addition, in this embodiment, the "unit" may include, for example, hardware resources implemented by a circuit in a broad sense and information processing of software that can be specifically realized by these hardware resources. Also, in this embodiment, various types of information are handled, and these information are represented, for example, by physical values of signal values representing voltage and current, the high and low of signal values as a set of binary bits composed of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculation can be executed on a circuit in a broad sense.

[0012] In addition, a circuit in a broad sense is a circuit realized by combining a circuit, circuitry, a processor, a memory, etc. as needed. That is, it includes an Application Specific Integrated Circuit (ASIC), programmable logic devices (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)), etc.

[0013] 1. Hardware Configuration In this section, the hardware configuration will be described.

[0014] <Construction Machine 1> FIG. 1 is a schematic diagram showing Construction Machine 1. Specifically, Construction Machine 1 in FIG. 1 is a pile driver. Construction Machine 1 includes a camera adjustment system (information processing device) 2, a main body 3, a camera group 4, and an inclination sensor 5.

[0015] The pile driver in this embodiment, particularly a large construction machine typified by a large pile driver, is disassembled and transported to the site due to transportation mass limitations, and assembly is performed at the site. Therefore, there is a possibility that the mounting posture of the camera may change. Even if it is calibrated in advance in the storage yard before transportation to the site, calibration is still required after reassembly. Also, if the camera cannot be easily calibrated by on-site workers assembling the large construction machine or the operator of the construction machine, it will be inconvenient and the man-hours for on-site work will increase. In addition, during on-site work, the camera may be damaged due to troubles during work, etc., and there is also a possibility of replacing the camera. Therefore, if there is a function to automatically calibrate the camera, there is no need to send a technician for calibration, and convenience is improved.

[0016] The main body 3 has a traveling body 31, a slewing body 32, a leader 33, an auger 34, and a lower guide 35. The traveling body 31 is a part that travels by crawlers. The slewing body 32 is configured to be slewed horizontally with respect to the traveling body 31. The slewing body 32 has a cabin 321, a house 322, outriggers 323 (hydraulic jacks), and a power frame 324. Further, a camera adjustment system 2, a camera group 4, and an inclination sensor 5 are arranged on the slewing body 32.

[0017] The leader 33 and the auger 34 together constitute a working device for driving piles. The leader 33 is attached in front of the slewing body 32 and stands upright in a direction of extending vertically when driving a pile. The auger 34 is a driving device for rotating a pile. The auger 34 is configured to be movable in the longitudinal direction of the leader 33. The lower guide 35 is connected to the leader 33 below the auger 34. A pile is inserted through the lower guide 35.

[0018] FIG. 2 is a plan view showing an example of the arrangement of the camera group 4. The camera group 4 captures images around the main body 3. The camera group 4 includes a first camera 41, a second camera 42, a third camera 43, a fourth camera 44, a fifth camera 45, a sixth camera 46, and a seventh camera 47 installed on the main body 3 (slewing body 32) of the construction machine 1. The first camera 41, the second camera 42, the third camera 43, the fourth camera 44, the fifth camera 45, the sixth camera 46, and the seventh camera 47 are arranged in this order counterclockwise in a plan view along the slewing direction of the slewing body 32.

[0019] Specifically, the first camera 41 is attached to the outrigger 323 at the left front of the revolving body 32. The second camera 42 is attached to the left side surface of the revolving body 32 (the outer surface of the left housing 322). The third camera 43 is arranged above the power supply frame 324 in a direction of photographing the left rear of the revolving body 32. The fourth camera 44 is arranged above the power supply frame 324 in a direction of photographing the rear of the revolving body 32. The fifth camera 45 is arranged above the power supply frame 324 in a direction of photographing the right rear of the revolving body 32. The sixth camera 46 is attached to the right side surface of the revolving body 32 (the outer surface of the right housing 322). The seventh camera 47 is attached to the outrigger 323 at the right front of the revolving body 32.

[0020] Each camera is arranged at a height at which a marker M arranged around the construction machine 1 can be photographed. The marker M is arranged, for example, at a position separated from the revolving body 32 by a turning radius R (for example, 5000 mm) or more, and the height is, for example, 200 mm. Each camera has an angle of view that can simultaneously photograph at least one marker M and a feature (such as the outer surface of the housing 322, the outrigger 323, etc.) indicating the position of the camera on the revolving body 32.

[0021] Two adjacent cameras are arranged such that their photographing ranges overlap. That is, the photographing ranges of the first camera 41 and the second camera 42 overlap each other. Similarly, the photographing range of the third camera 43 overlaps with that of the second camera 42. The same applies to the combinations of the fourth camera 44 and the fifth camera 45, the fifth camera 45 and the sixth camera 46, the sixth camera 46 and the seventh camera 47, and the seventh camera 47 and the first camera 41.

[0022] FIG. 3 is a schematic diagram of the adjustment mechanism for the orientation of the first camera 41. The orientation of the first camera 41 is controlled by a first servo motor 401A, a second servo motor 401B, and a third servo motor 401C.

[0023] The first servo motor 401A is a servo motor that changes the roll angle of the first camera 41, and rotates the second servo case 402B, the turntable 402C, and the first camera 41 around the roll axis P1. The first servo motor 401A is housed in the first servo case 402A and fixed to the wall surface of the first servo case 402A. The first servo case 402A is fixed to the swivel body 32 (the outer surface of the housing 322, the outrigger 323, etc.).

[0024] The second servo motor 401B is a servo motor that changes the yaw angle of the first camera 41, and rotates the turntable 402C and the first camera 41 around the yaw axis P2. The second servo motor 401B is housed in the second servo case 402B and fixed to the wall surface of the second servo case 402B. The second servo case 402B is fixed to the rotation axis of the first servo motor 401A.

[0025] The third servo motor 401C is a servo motor that changes the pitch angle of the first camera 41, and rotates the first camera 41 around the pitch axis P3. The third servo motor 401C is fixed to the turntable 402C. The turntable 402C is fixed to the rotation axis of the second servo motor 401B. The rotation axis of the third servo motor 401C is fixed to the camera mounting stay 403 that holds the first camera 41.

[0026] The tilt sensor 5 is configured to detect the tilt of the main body 3 with respect to the horizontal direction. As the tilt sensor 5, for example, an inertial measurement unit (IMU) that detects three-axis angular velocity and acceleration can be used.

[0027] <Camera adjustment system 2> The camera adjustment system 2 adjusts the orientation of the camera group 4. The camera adjustment system 2 of the construction machine consists of one or more information processing devices or components. Hereinafter, these components will be described.

[0028] Figure 4 is a block diagram showing the hardware configuration of the camera adjustment system 2. The camera adjustment system 2 includes a communication bus 20, a communication unit 21, a storage unit 22, and a processor 23. The communication unit 21, the storage unit 22, and the processor 23 are electrically connected via the communication bus 20 inside the camera adjustment system 2.

[0029] <Communication unit 21> Although wired communication means such as USB, IEEE1394, Thunderbolt (registered trademark), and wired LAN network communication are preferred for the communication unit 21, wireless LAN network communication, mobile communication such as 3G / LTE / 5G, and BLUETOOTH (registered trademark) communication may be included as necessary. That is, it is more preferable to implement it as a set of these multiple communication means. That is, the camera adjustment system 2 may communicate various information from the outside via the communication unit 21 and the network.

[0030] <Storage unit 22> The storage unit 22 stores various information defined by the foregoing description. This can be implemented as a storage device such as a solid state drive (SSD) that stores various programs related to the camera adjustment system 2 executed by the processor 23, or as a memory such as a random access memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to the calculation of the program. The storage unit 22 stores various programs, variables, etc. related to the camera adjustment system 2 executed by the processor 23.

[0031] <Processor 23> The processor 23 processes and controls the overall operations related to the camera adjustment system 2. The processor 23 is, for example, a Central Processing Unit (CPU). The processor 23 realizes various functions related to the camera adjustment system 2 by reading a predetermined program stored in the storage unit 22. That is, the information processing by software stored in the storage unit 22 is specifically realized by the processor 23, which is an example of hardware, and can be executed as each functional unit included in the processor 23. These will be described in more detail in the next section. Note that the processor 23 is not limited to being single, and may be implemented to have a plurality of processors 23 for each function, or a combination thereof may also be used.

[0032] The camera adjustment system 2 may be in an on-premises form or a cloud form. As the cloud form of the camera adjustment system 2, for example, it may provide the above-described functions and processes in the form of Software as a Service (SaaS) or cloud computing.

[0033] 2. Functional Configuration In this section, the functional configuration of this embodiment will be described. The information processing by software stored in the storage unit 22 is specifically realized by the processor 23, which is an example of hardware, and can be executed as each functional unit included in the processor 23.

[0034] FIG. 5 is a block diagram showing the functions realized by the camera adjustment system 2 (processor 23). Specifically, the camera adjustment system 2 (processor 23) includes an inclination information acquisition unit 231, a correction value calculation unit 232, and a camera correction unit 233.

[0035] <Inclination Information Acquisition Unit 231> The inclination information acquisition unit 231 acquires inclination information from the inclination sensor 5. The inclination information includes the inclination angle of the main body 3 with respect to the horizontal direction.

[0036] <Correction value calculation unit 232> The correction value calculation unit 232 calculates a correction value for the inclination of the construction machine 1 based on the inclination information acquired by the inclination information acquisition unit 231. This correction value is referred to when correcting the orientation of the camera.

[0037] <Camera correction unit 233> The camera correction unit 233 corrects the orientation of the first camera 41 or the second camera 42 while referring to the correction value calculated by the correction value calculation unit 232 so that the overlap amount between the first correction image acquired from the first camera 41 and the second correction image acquired from the second camera 42 becomes a certain value or more.

[0038] Specifically, the first correction image and the second correction image acquired by the camera correction unit 233 include a common marker. That is, as shown in FIG. 2, the first camera 41 and the second camera 42 photograph the marker M from different positions. Therefore, the first correction image and the second correction image include the marker M photographed from different directions.

[0039] The marker M is not particularly limited as long as it has a shape and color that are easy to extract (recognize) from the image photographed by the camera. For example, a checkerboard may be used as the marker M.

[0040] The camera correction unit 233 generates a composite image by combining the first correction image and the second correction image, and obtains a transformation matrix for converting the first correction image or the second correction image into the composite image using the marker included in both the first correction image and the second correction image as a feature point, and changes the direction of the first camera 41 or the second camera 42 based on the transformation matrix. Thereby, the orientation of the camera can be corrected so that the degree of overlap of the markers in the first correction image and the second correction image becomes high.

[0041] Specifically, first, the camera correction unit 233 overlays the entire first correction image and the entire second correction image so that the overlapping area between the marker included in the first correction image and the marker included in the second correction image is equal to or greater than a predetermined first threshold value, and generates a composite image. Note that the marker in each image is discriminated by pattern matching between the shape of the marker stored in advance and the shape corrected by the correction value for the inclination of the construction machine 1 calculated by the correction value calculation unit 232.

[0042] The transformation matrix is a matrix that corrects the deviation of the first correction image or the second correction image with respect to the composite image. As a procedure for obtaining the transformation matrix, for example, the DLT algorithm can be used. The transformation matrix is obtained one by one for each of the first correction image and the second correction image.

[0043] The DLT algorithm obtains a homography matrix H that satisfies P' = HP from a given set of n (≧4) pairs of two-dimensional corresponding points P i and P i' and. Specifically, first, a matrix A represented by the following formula (1) i ' = HP i is obtained. The matrix A i is a 2-row 9-column matrix composed of two linearly independent expressions among the matrices obtained from the set of corresponding points. In image conversion, x i , y i are the positions (coordinates) on the x-axis and y-axis in each image, and z i is 1. i is 1.

Equation

[0044] Next, from the n obtained A i , a matrix A represented by the following formula (2) is obtained. The matrix A is a 2n-row 9-column matrix.

Equation

[0045] For the obtained matrix A, a homography matrix H with 2n rows and 9 columns is obtained by singular value decomposition of the following equation (3). In equation (3), U is a unitary matrix with 2n rows and 2n columns, and V * is the adjoint matrix of the unitary matrix V with 9 rows and 9 columns.

Number

[0046] The camera correction unit 233 uses the DLT algorithm to set the feature points acquired from the markers included in the composite image as P i , and the points corresponding to P i in the first correction image or the second correction image as P i ', and sets the obtained homography matrix H as a transformation matrix. Note that when acquiring the feature points (markers) in the image, the correction value for the inclination of the construction machine 1 calculated by the correction value calculation unit 232 is referenced.

[0047] The camera correction unit 233 calculates a reprojection error between the first correction image and the second correction image with respect to the composite image. This reprojection error reflects the degree of overlap of the markers in the first correction image and the second correction image, and the smaller the reprojection error, the greater the degree of overlap of the markers. Specifically, the camera correction unit 233 calculates a reprojection error ε by the following formula (4) using the transformation matrices (homography matrix H) of the obtained first correction image and second correction image. The reprojection error ε indicates the magnitude of positional deviation between corresponding feature points of the first correction image and the second correction image after transformation using the homography matrix H.

Number

[0048] Instead of the reprojection error, the amount of deviation between the first and second correction images obtained by the RANSAC algorithm (the degree of overlap of the markers) may be used.

[0049] If the reprojection error does not exceed a predetermined second threshold value, the camera correction unit 233 determines that it is not necessary to change the direction of the first camera 41 (when comparing the first correction image and the composite image) and / or the second camera 42 (when comparing the second correction image and the composite image).

[0050] On the other hand, if the reprojection error exceeds the second threshold value, the camera correction unit 233 determines whether the overlap amount between the first correction image and the second correction image is equal to or greater than a predetermined third threshold value. The overlap amount is the area or the number of pixels of the overlapping region (common region) between the first correction image and the second correction image in the composite image. When calculating the overlap amount, the correction value for the inclination of the construction machine 1 calculated by the correction value calculation unit 232 is referred to.

[0051] When the reprojection error exceeds a certain range (the second threshold value) and the overlap amount between the first correction image and the second correction image is less than a certain value (the third threshold value), the camera correction unit 233 changes the direction of the first camera 41 or the second camera 42 based on the transformation matrix. Thereby, while eliminating the influence of noise in each image, the orientation of the camera can be adjusted so that the overlap amount between the first correction image and the second correction image becomes a certain value or more.

[0052] The direction (viewing angle) of the first camera 41 or the second camera 42 is changed by driving (rotating) the first servo motor 401A, the second servo motor 401B, and the third servo motor 401C. The rotation angles (roll angle, pitch angle, and yaw angle) of each servo motor are calculated based on the transformation matrix. That is, the rotation angle of each servo motor is derived from the relationship (rotation angle) between the feature points of the first correction image (or the second correction image) before transformation and the feature points in the image after transformation (projective transformation) by the transformation matrix.

[0053] The camera correction unit 233 outputs a PWM signal corresponding to the target rotation angle (actuation amount) to each of the first servo motor 401A, the second servo motor 401B, and the third servo motor 401C. Thereby, the viewing angle of the first camera 41 or the second camera 42 is corrected so that the reprojection error between the first correction image and the second correction image becomes small.

[0054] Further, when the reprojection error exceeds a certain range and the overlap amount between the first correction image and the second correction image is equal to or greater than a certain value (third threshold value), after converting the first correction image and the second correction image with the transformation matrix, the reprojection error is calculated again using the converted first correction image and second correction image. Thereby, it is possible to determine whether correction of the camera direction is necessary in a state where the influence of noise included in the image is excluded.

[0055] Specifically, when the reprojection error exceeds a regulation (for example, the second threshold value) but the overlap amount satisfies the regulation (for example, the third threshold value), the camera correction unit 233 converts the first correction image and the second correction image with the calculated transformation matrix without changing the direction of the camera. The conversion of the first correction image is performed with the transformation matrix obtained from the composite image and the first correction image, and the conversion of the second correction image is performed with the transformation matrix obtained from the composite image and the second correction image.

[0056] The camera correction unit 233 may obtain a matrix of affine transformation instead of a homography matrix as a transformation matrix for converting the first correction image and the second correction image into a composite image. In this case, the reprojection error is obtained from the difference in coordinates of corresponding points (feature points) between the first correction image and the second correction image after affine transformation by the matrix. Further, when the overlap amount is equal to or greater than a certain value, the camera correction unit 233 performs affine transformation on the first correction image and the second correction image using the matrix.

[0057] The camera correction unit 233 creates a composite image again using at least one of the first corrected image and the second corrected image that have been converted, and calculates the reprojection error. After calculating the reprojection error, the camera correction unit 233 determines again whether or not the reprojection error exceeds the specified value as described above.

[0058] After correcting the orientation of the first camera 41 or the second camera 42, the camera correction unit 233 acquires the first corrected image and the second corrected image again, and calculates the reprojection error again. Thereby, the adjustment of the camera can be automatically repeated until the overlap amount becomes an appropriate value.

[0059] Specifically, the camera correction unit 233 creates a composite image again using the newly acquired first corrected image and second corrected image, and calculates the reprojection error. After calculating the reprojection error, the camera correction unit 233 determines again whether or not the reprojection error exceeds the specified value as described above.

[0060] After adjusting the second camera 42 using the first corrected image and the second corrected image, the camera correction unit 233 further corrects the orientation of the third camera 43 while referring to the correction value calculated by the correction value calculation unit 232 so that the overlap amount between the second corrected image acquired from the second camera 42 and the third corrected image acquired from the third camera 43 becomes a certain value or more. Thereby, since the third camera 43 can be adjusted using the second camera 42 that is the target of adjustment, the adjustment accuracy of the camera group 4 is improved.

[0061] The reprojection error between the second corrected image and the third corrected image is calculated in the same procedure as the reprojection error between the first corrected image and the second corrected image. Also, the correction of the orientation of the third camera 43 using the second corrected image and the third corrected image is executed in the same procedure as the correction of the orientation of the second camera 42 using the first corrected image and the second corrected image.

[0062] After correcting the orientation of the second camera 42 and before acquiring the third correction image, the camera correction unit 233 rotates the slewing body 32 on which the first camera 41, the second camera 42, and the third camera 43 of the construction machine 1 are installed so that the acquired second correction image and third correction image include the marker M used for correcting the orientation of the first camera 41 or the second camera 42 (that is, included in the first correction image and the second correction image). Thereby, without using a plurality of markers M as shown in FIG. 2, the correction of a plurality of camera sets can be performed by one marker M.

[0063] For the remaining camera sets of the construction machine 1 (the set of the third camera 43 and the fourth camera 44, the set of the fourth camera 44 and the fifth camera 45, the set of the fifth camera 45 and the sixth camera 46, the set of the sixth camera 46 and the seventh camera 47, and the set of the seventh camera 47 and the first camera 41), the orientation correction is also performed in the same procedure. The correction of these camera sets is performed, for example, in a counterclockwise order. That is, the orientation of the second camera 42 is corrected by the set of the first camera 41 and the second camera 42, and the orientation of the third camera 43 is corrected by the set of the second camera 42 and the third camera 43... By repeating this procedure until the orientation of the seventh camera 47 is corrected, the calibration of all the cameras is performed.

[0064] 3. Camera adjustment method In this section, a camera adjustment method using the camera adjustment system 2 will be described. In this camera adjustment method, each part of the camera adjustment system 2 is executed by a computer as each step.

[0065] Specifically, the camera adjustment method includes an inclination information acquisition step, a correction value calculation step, and a camera correction step.

[0066] In the tilt information acquisition step, tilt information is acquired from the tilt sensor 5 provided in the construction machine 1. In the correction value calculation step, a correction value for the tilt of the construction machine 1 is calculated based on the tilt information. In the camera correction step, while referring to the correction value, the orientation of the first camera 41 or the second camera 42 is corrected so that the overlap amount between the first correction image acquired from the first camera 41 and the second correction image acquired from the second camera 42 becomes a certain value or more. In the camera correction step, further, while referring to the correction value, the orientation of the second camera 42 or the third camera 43 is corrected so that the overlap amount between the second correction image and the third correction image acquired from the third camera 43 becomes a certain value or more.

[0067] FIG. 6 is a flowchart showing the flow of the camera adjustment process executed by the camera adjustment system 2. First, the camera adjustment system 2 zero-corrects the servo motors of each camera (step S110). Next, the camera adjustment system 2 acquires tilt information from the tilt sensor 5 (step S120).

[0068] After the acquisition of the tilt information, the camera adjustment system 2 executes the camera orientation correction process (step S130). FIG. 7 is a flowchart showing the flow of the camera orientation correction process. In the orientation correction process, the camera adjustment system 2 first acquires images from two cameras that form a pair (for example, the first camera 41 and the second camera 42) (step S210). After the image acquisition, the camera adjustment system 2 synthesizes the images of the two cameras (for example, the first correction image and the second correction image) (step S220).

[0069] After the image synthesis, the camera adjustment system 2 calculates the reprojection error of the camera's image with respect to the synthesized image (step S230). Specifically, the reprojection error is calculated by obtaining the transformation matrix of the camera's image with respect to the synthesized image. Next, the camera adjustment system 2 determines whether the calculation of the reprojection error was successful (step S240). The success of the reprojection error calculation means that the transformation matrix and the reprojection error were calculated within a certain period of time. If the reprojection error cannot be calculated within a certain period of time (S240: NO), the camera adjustment system 2 terminates the camera orientation correction process while setting an error flag. On the other hand, if the reprojection error can be calculated within a certain period of time (S240: YES), the camera adjustment system 2 determines whether the calculated reprojection error exceeds a second threshold value (step S250).

[0070] If the reprojection error does not exceed the second threshold value (S250: NO), the camera adjustment system 2 terminates the camera orientation correction process while setting a normal flag. On the other hand, if the reprojection error exceeds the second threshold value (S250: YES), the camera adjustment system 2 determines whether the overlap amount of the images of the two cameras is equal to or greater than a third threshold value (step S260). If the overlap amount is equal to or greater than the third threshold value (S260: YES), the camera adjustment system 2 applies the calculated transformation matrix to the camera's image (step S270) and repeats the steps from image synthesis (S220).

[0071] On the other hand, if the overlap amount is less than the third threshold value (S260: NO), the camera adjustment system 2 calculates the actuation amount of the servo motor for correcting the camera's orientation (step S280). After calculating the actuation amount, the camera adjustment system 2 determines whether the calculated actuation amount is within the range in which the servo motor can actually be actuated (step S290).

[0072] When the amount of actuation is not within the range where actuation is possible (S290: NO), the camera adjustment system 2 ends the camera orientation correction process while setting an error flag. On the other hand, when the amount of actuation is within the range where actuation is possible (S290: YES), the camera adjustment system 2 actuates the camera's servo motor (step S300) and repeats the steps from image capturing (S210).

[0073] After the completion of the camera orientation correction process, as shown in FIG. 6, the camera adjustment system 2 determines the presence or absence of an error flag (step S140). When the error flag is set (S140: YES), the camera adjustment system 2 abnormally terminates the camera adjustment process. In this case, for example, measures such as manually adjusting the camera orientation are taken. On the other hand, when the error flag is not set (the normal flag is set) (S140: NO), the camera adjustment system 2 determines whether there is a set of cameras for which the camera orientation correction process has not been performed (step S150).

[0074] When there is a set of unprocessed cameras (S150: YES), the camera adjustment system 2 rotates the rotating body 32 as necessary to change the camera's shooting range (step S160) and repeats the steps from the camera orientation correction process (S130). On the other hand, when there is no set of unprocessed cameras (S150: NO), the camera adjustment system 2 normally terminates the camera adjustment process. As a result, the camera orientation correction process is performed for all sets of cameras.

[0075] The error flag when the reprojection error cannot be calculated (S240) and the error flag when the range where actuation is possible is not satisfied (S290) may be set separately. When the error flag is set (S140), the camera adjustment system 2 may output an error corresponding to the individual flag (for example, display it on a display or the like). Also, when checking the error flag (S140), the camera adjustment system 2 may count the cumulative number of occurrences of the error flag and output an error corresponding to the count.

[0076] For example, the camera adjustment system 2 may output an error display including the number of the camera that could not be actuated and the countermeasure, such as "The first camera cannot be actuated. Please adjust it manually." Further, the camera adjustment system 2 may output a system error message, such as "The number of trials of the camera adjustment system has exceeded the specified value. Alignment cannot be performed correctly. Please contact the manufacturer."

[0077] 4. Operation Using the overlap amount between the first correction image and the second correction image, the orientation of the first camera 41 or the second camera 42 can be automatically corrected. Further, since the inclination of the construction machine 1 is considered when correcting the orientation of the camera, calibration of a plurality of cameras can be performed with high accuracy.

[0078] As described above, the embodiments of the present invention have been described, but the present invention is not limited thereto, and can be appropriately changed without departing from the technical idea of the invention.

[0079] 5. Others The aspect of this embodiment is not limited to the camera adjustment system 2 of the construction machine, and may be a camera adjustment method or a program of the construction machine. The camera adjustment method of the construction machine includes each step of the camera adjustment system 2 of the construction machine. The program causes a computer to function as the camera adjustment system 2 of the construction machine.

[0080] The construction machine 1 is not limited to a pile driver. That is, the camera adjustment system 2 is also applicable to construction machines other than pile drivers.

[0081] The camera adjustment system 2 does not necessarily have to be installed on the construction machine 1. For example, the camera adjustment system 2 may be configured by an information processing device installed in a facility outside the construction machine 1.

[0082] It may also be provided in each of the aspects described below.

[0083] (1) A camera adjustment system for construction machinery, comprising a processor, wherein the processor is configured to execute an inclination information acquisition step, a correction value calculation step, and a camera correction step. In the inclination information acquisition step, inclination information is acquired from an inclination sensor provided in the construction machinery. In the correction value calculation step, a correction value for the inclination of the construction machinery is calculated based on the inclination information. In the camera correction step, while referring to the correction value, the orientation of the first camera or the second camera is corrected so that the overlap amount between a first correction image acquired from the first camera and a second correction image acquired from the second camera becomes a certain value or more. Here, the first camera and the second camera are installed on the main body of the construction machinery, and the shooting ranges thereof overlap with each other. A camera adjustment system for construction machinery.

[0084] (2) In the camera adjustment system for construction machinery according to (1) above, in the camera correction step, further, while referring to the correction value, the orientation of the third camera is corrected so that the overlap amount between the second correction image and a third correction image acquired from the third camera becomes a certain value or more. Here, the third camera is installed on the construction machinery, and the shooting range thereof overlaps with that of the second camera. A camera adjustment system for construction machinery.

[0085] (3) In the camera adjustment system for construction machinery according to (2) above, the first correction image and the second correction image acquired in the camera correction step include a common marker. In the camera correction step, after correcting the orientation of the second camera and before acquiring the third correction image, the revolving body of the construction machinery on which the first camera, the second camera, and the third camera are installed is rotated so that the acquired second correction image and the third correction image include the marker. A camera adjustment system for construction machinery.

[0086] (4) In the camera adjustment system for a construction machine according to any one of (1) to (3) above, in the camera correction step, a composite image is generated by synthesizing the first correction image and the second correction image, and a marker included in both the first correction image and the second correction image is used as a feature point to obtain a transformation matrix for converting the first correction image or the second correction image into the composite image, and the direction of the first camera or the second camera is changed based on the transformation matrix. A camera adjustment system for a construction machine.

[0087] (5) In the camera adjustment system for a construction machine according to (4) above, in the camera correction step, a reprojection error between the first correction image and the second correction image with respect to the composite image is calculated, and when the reprojection error exceeds a certain range and the overlap amount is less than a certain value, the direction of the first camera or the second camera is changed based on the transformation matrix. A camera adjustment system for a construction machine.

[0088] (6) In the camera adjustment system for a construction machine according to (5) above, in the camera correction step, when the reprojection error exceeds a certain range and the overlap amount is equal to or more than a certain value, after converting the first correction image and the second correction image with the transformation matrix, the reprojection error is calculated again using the converted first correction image and the second correction image. A camera adjustment system for a construction machine.

[0089] (7) In the camera adjustment system for a construction machine according to (5) or (6) above, in the camera correction step, after correcting the orientation of the first camera or the second camera, the first correction image and the second correction image are acquired again, and the reprojection error is calculated again. A camera adjustment system for a construction machine.

[0090] (8) A construction machine comprising an information processing device as the camera adjustment system for a construction machine according to any one of (1) to (7) above.

[0091] (9) A method for adjusting a camera of a construction machine, the method comprising each step of the camera adjustment system of the construction machine according to any one of (1) to (7) above.

[0092] (10) A program for causing a computer to function as the camera adjustment system of the construction machine according to any one of (1) to (7) above. Of course, this is not the limit.

[0093] Finally, although various embodiments according to the present disclosure have been described, these are presented as examples and are not intended to limit the scope of the invention. The novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. The embodiments and their modifications are included in the scope and gist of the invention and are included in the invention described in the claims and the equivalent scope thereof.

Explanation of Reference Numerals

[0094] 1: Construction machine 2: Camera adjustment system 3: Main body 4: Camera group 5: Inclinometer 20: Communication bus 21: Communication unit 22: Storage unit 23: Processor 31: Traveling body 32: Slewing body 33: Leader 34: Auger 35: Lower guide 41: First camera 42: Second camera 43: Third camera 44: Fourth camera 45: Fifth camera 46: Sixth camera 47: Seventh camera 231: Inclination information acquisition unit 232: Correction value calculation unit 233: Camera correction unit 321: Cabin 322: House 323: Outrigger 324: Power frame 401A: First servo motor 401B: Second servo motor 401C: Third servo motor 402A: First servo case 402B: Second servo case 402C: Turntable 403: Camera mounting stage

Claims

1. A camera adjustment system for a construction machine, comprising: A processor is provided. The processor is configured to execute a tilt information acquisition step, a correction value calculation step, and a camera correction step; In the inclination information acquisition step, inclination information is acquired from an inclination sensor provided in the construction machine, In the correction value calculation step, a correction value for the inclination of the construction machine is calculated based on the inclination information, In the camera correction step, the orientation of the first camera or the second camera is corrected while referring to the correction value so that the amount of overlap between the first correction image acquired from the first camera and the second correction image acquired from the second camera is equal to or greater than a certain value, wherein the first camera and the second camera are installed on the main body of the construction machine and have overlapping shooting ranges.This is a camera adjustment system for construction machinery.

2. 2. The camera adjustment system for a construction machine according to claim 1, In the camera correction step, the orientation of the third camera is further corrected while referring to the correction value so that the amount of overlap between the second correction image and the third correction image obtained from the third camera is equal to or greater than a certain value, wherein the third camera is installed on the construction machine and has an overlapping shooting range with the second camera.A camera adjustment system for construction machinery.

3. 3. The camera adjustment system for a construction machine according to claim 2, the first correction image and the second correction image acquired in the camera correction step include a common marker, In the camera correction step, after correcting the orientation of the second camera and before acquiring the third correction image, a rotating body of the construction machine on which the first camera, the second camera, and the third camera are installed is rotated so that the marker is included in the acquired second correction image and the third correction image, in a camera adjustment system for a construction machine.

4. 2. The camera adjustment system for a construction machine according to claim 1, In the camera correction step, a composite image is generated by combining the first correction image and the second correction image, and a transformation matrix is ​​calculated for converting the first correction image or the second correction image into the composite image using markers contained in both the first correction image and the second correction image as feature points, and the direction of the first camera or the second camera is changed based on the transformation matrix, in a camera adjustment system for a construction machine.

5. The camera adjustment system for a construction machine according to claim 4, In the camera correction step, a reprojection error between the first correction image and the second correction image relative to the composite image is calculated, and if the reprojection error exceeds a certain range and the overlap amount is less than a certain value, the direction of the first camera or the second camera is changed based on the transformation matrix.This is a camera adjustment system for construction machinery.

6. 6. The camera adjustment system for a construction machine according to claim 5, In the camera correction step, if the reprojection error exceeds a certain range and the overlap amount is equal to or greater than a certain value, the first correction image and the second correction image are transformed with the transformation matrix, and then the reprojection error is calculated again using the transformed first correction image and the second correction image.This is a camera adjustment system for construction machinery.

7. The camera adjustment system for a construction machine according to claim 5, In the camera correction step, after correcting the orientation of the first camera or the second camera, the first correction image and the second correction image are newly acquired and the reprojection error is recalculated.

8. A construction machine comprising an information processing device as the camera adjustment system for the construction machine according to claim 1.

9. A camera adjustment method for a construction machine, comprising: A method for adjusting a camera of a construction machine, comprising the steps of the camera adjustment system for a construction machine according to any one of claims 1 to 7.

10. A program, A program for causing a computer to function as the camera adjustment system for a construction machine according to any one of claims 1 to 7.

Images