Apparatus and method for camera calibration using pattern pair

Abstract

The present invention relates to a vehicle camera calibration technology applicable to advanced driver assistance systems (ADAS) and autonomous driving systems, and more specifically, to a camera calibration device and method using pattern pairs that can precisely and efficiently calibrate multiple cameras even under various environmental constraints such as the type and size of the vehicle and the installation location of the cameras.

Description

Technology Field

[0001] The present invention relates to a vehicle camera calibration technology applicable to advanced driver assistance systems (ADAS) and autonomous driving systems, and more specifically, to a camera calibration device and method using pattern pairs that can precisely and efficiently calibrate multiple cameras even under various environmental constraints such as the type and size of the vehicle and the installation location of the cameras. Background Technology

[0003] Recently, the importance of systems that utilize multiple cameras mounted on vehicles to perceive the surrounding environment in 360 degrees and determine driving conditions (e.g., Around View Monitoring (AVM) systems, autonomous driving systems, construction site safety systems) is rapidly increasing.

[0004] The performance of such a system is determined by camera calibration technology that precisely aligns multiple images input from each camera into a single unified three-dimensional spatial coordinate system. Generally, camera calibration involves the process of accurately estimating the intrinsic parameters (focal length, principal point, lens distortion coefficient, etc.) and extrinsic parameters (the three-dimensional position and orientation of the camera) of each camera.

[0005] A typical multi-camera calibration method places a single calibration pattern, such as a checkerboard or circular dot pattern, in the common field of view where all cameras can capture the same area, and estimates camera parameters by analyzing the images captured by each camera. This approach can be effectively applied in environments where vehicle shapes are relatively standardized—such as passenger cars or standardized commercial trucks—where camera installation locations are predictable and it is easy to secure an area where multiple cameras can commonly capture the pattern.

[0006] However, in the case of special purpose vehicles or industrial heavy equipment such as wheel loaders, excavators, pile drivers, large cranes, and agricultural tractors, the size and shape of the vehicles vary greatly, and they often have asymmetrical or irregular structures. As a result, the installation position of the camera is inevitably irregular, and above all, there is a problem in that it is difficult to apply conventional single-pattern-based calibration methods because there is no common area where multiple cameras can simultaneously capture a single common pattern, or such an area is extremely narrow.

[0007] In such environments, calibrating each camera individually and integrating the results is not only highly complex and challenging, but also faces limitations in guaranteeing the accuracy and robustness of the final calibration. Consequently, calibration remains a major technical challenge in the development of multi-camera systems for special-purpose vehicles, and the development of technologies to address this is required. The problem to be solved

[0009] The present invention has been devised to solve the problems of the aforementioned prior art, and

[0010] The purpose is to successfully perform camera calibration with strong robustness and excellent accuracy, even in environments where there is no common shooting area among multiple cameras or where it is difficult to secure one.

[0011] In addition, the present invention aims to provide a flexible and highly scalable camera calibration device and method capable of performing multi-camera system calibration even in special vehicles or heavy equipment with highly irregular sizes and shapes, such as wheel loaders, excavators, and special transport equipment. means of solving the problem

[0013] A device for calibrating a first camera and a second camera installed in a vehicle using a pattern pair comprising a first pattern and a second pattern having a prior relative positional relationship between them, comprising: an image input unit receiving, respectively, a first image and a second image capturing part or all of the pattern pair from the first camera and the second camera; a pattern detection unit detecting a pattern region in each of the first image and the second image; a pattern verification unit determining whether the pattern detected in the first image and the pattern detected in the second image are the same pattern captured by both cameras in common or different patterns constituting the pattern pair; and a coordinate system integration unit integrating the coordinate systems of the first camera and the second camera using the prior relative positional relationship of the pattern pair when the result of the determination by the pattern verification unit determines that the detected patterns are different patterns.

[0014] In addition, in one embodiment, the coordinate system integration unit calculates a relative positional relationship (T_C1_P1) between the 3D coordinate system of the first camera and the first pattern, calculates a relative positional relationship (T_C2_P2) between the 3D coordinate system of the second camera and the second pattern, and integrates the coordinate systems of the first camera and the second camera by calculating a relative transformation relationship between the 3D coordinate system of the first camera and the 3D coordinate system of the second camera by connecting a predefined relative positional relationship (T_P1_P2) between the first pattern and the second pattern and the calculated (T_C1_P1 and T_C2_P2).

[0015] In addition, in one embodiment, the coordinate system integration unit integrates the coordinate systems of the first camera and the second camera based on the same pattern when the judgment result of the pattern verification unit is that the detected patterns are the same pattern.

[0016] In addition, in one embodiment, the first pattern and the second pattern have different sizes and shapes.

[0017] In addition, in one embodiment, the pattern detection unit detects a pattern region using at least one image processing algorithm among edge detection, corner detection, feature point extraction and matching, color and brightness-based segmentation, or deep learning-based object detection.

[0018] In addition, in one embodiment, the pattern verification unit determines whether the pattern is the same by comparing feature point descriptors of the detected pattern or verifying a unique ID.

[0019] Meanwhile, a method for calibrating a first camera and a second camera installed in a vehicle using a pattern pair comprising a first pattern and a second pattern with a predefined relative positional relationship between them comprises: (a) a step of acquiring a first image and a second image by capturing part or all of the pattern pair with the first camera and the second camera; (b) a step of detecting a pattern region in each of the acquired first image and the second image; (c) a step of determining whether the pattern detected in the first image and the pattern detected in the second image are the same pattern captured in common by the two cameras or different patterns constituting the pattern pair; and (d) a step of integrating the coordinate systems of the two cameras by calculating the relative positional and directional relationship of the first camera and the second camera using the predefined relative positional relationship of the pattern pair when, as a result of the determination in step (c), the detected patterns are determined to be different patterns.

[0020] Additionally, in one embodiment, the step (d) further comprises: (d1) a step of calculating a relative positional relationship between the three-dimensional coordinate system of the first camera and the first pattern; (d2) a step of calculating a relative positional relationship between the three-dimensional coordinate system of the second camera and the second pattern; and (d3) a step of calculating a relative transformation relationship between the three-dimensional coordinate system of the first camera and the three-dimensional coordinate system of the second camera together with the relative positional relationships calculated in steps (d1) and (d2), using a predefined relative positional relationship between the first pattern and the second pattern.

[0021] Additionally, in one embodiment, the method further includes the step of integrating the coordinate systems of the first camera and the second camera based on the same pattern when, as a result of the judgment of step (c), (c1) the detected patterns are determined to be the same pattern.

[0022] In addition, in one embodiment, step (c) determines whether the pattern is the same by comparing feature point descriptors of the detected pattern or verifying a unique ID. Effects of the invention

[0024] The present invention overcomes the limitations of existing camera calibration technology and provides the following effects.

[0025] The present invention has the advantage of being applicable to large and irregularly shaped special vehicles and heavy equipment, such as wheel loaders, excavators, cranes, and agricultural machinery, because it enables calibration by capturing parts of pattern pairs (different patterns) through each camera even in environments where there is no common shooting area between multiple cameras and calibration was impossible with conventional methods.

[0026] In addition, the present invention has the advantage of being able to perform the calibration of the entire system stably and reliably by utilizing three-dimensional relative positional relationship information of pre-defined pattern pairs, even in situations where there are significantly fewer constraints on pattern placement, some patterns are obscured, or the field of view of a specific camera is extremely limited.

[0027] Furthermore, by utilizing information on a pair of patterns and the precise relative positional relationship between them in the calibration process, the present invention can incorporate more geometric constraints into the calibration model than conventional technology using only a single pattern, which provides the advantage of increasing the accuracy of the final calibration result.

[0028] Furthermore, the present invention offers the advantage of reducing the inconvenience of existing methods that require fixing a calibration pattern to a specific location or performing complex high-precision placement operations. In particular, the operator can flexibly place the pattern according to on-site conditions, enabling rapid and efficient calibration, which leads to the advantage of reducing maintenance and production costs. Brief explanation of the drawing

[0030] FIG. 1 is a block diagram schematically showing the system configuration of a camera calibration system according to a preferred embodiment of the present invention. FIG. 2 is a block diagram schematically showing the configuration of a calibration device according to a preferred embodiment of the present invention. FIG. 3 is a flowchart showing each step of a camera calibration method according to a preferred embodiment of the present invention. FIG. 4 is a flowchart showing the detailed steps of the camera coordinate system integration step according to a preferred embodiment of the present invention. Figure 5 is a conceptual diagram illustrating a scenario in which two cameras have a common shooting area and simultaneously capture the same calibration pattern. Figure 6 is a conceptual diagram illustrating a scenario in which two cameras each capture different patterns constituting a pattern pair, with no common shooting area at all or very narrow. Specific details for implementing the invention

[0031] Preferred embodiments of the present invention will be described in detail below with reference to the attached drawings. It is clearly stated that the present invention is not limited by the embodiments disclosed below but can be implemented in various forms and ways. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention. Throughout the drawings, the same reference numerals indicate the same components.

[0033] 1. System Configuration

[0034] FIG. 1 schematically shows the configuration of a camera calibration system according to a preferred embodiment of the present invention.

[0035] The calibration system of the present invention includes a vehicle (10), a plurality of cameras (20), a pair of patterns (30), and a calibration device (40).

[0037] 1.1. Vehicle (10)

[0038] In a preferred embodiment of the present invention, the vehicle (10) subject to calibration encompasses various types of heavy equipment or irregular vehicles, such as wheel loaders, excavators, pile drivers, drilling rigs, cranes, forklifts, large trucks, agricultural tractors, and special military vehicles. Unlike conventional passenger cars or general commercial vehicles, these vehicles have very diverse and complex sizes and shapes, and camera installation positions may be irregular, making it difficult or impossible to apply conventional single-pattern based calibration methods.

[0039] However, it should be understood that the vehicle (10) to which the present invention is applied is not limited to the special purpose vehicles listed above, and can be applied to any known vehicle.

[0041] 1.2. Camera (20)

[0042] The above camera is an image input device installed on the exterior of the vehicle (10) to capture the surrounding environment.

[0043] For convenience of explanation in a preferred embodiment of the present invention, the camera (20) is described as including a first camera (21) and a second camera (22), but the present invention is not limited thereto and can be extended to three or more multi-camera systems (e.g., 4-channel around view, multi-directional stereo vision system).

[0044] The above cameras (20), that is, the first camera (21) and the second camera (22), have their own internal parameters (focal length, principal point, lens distortion coefficient, etc.) and external parameters (position and orientation of the camera in three-dimensional space).

[0045] The calibration process of the present invention is to precisely calculate or estimate the parameters of the camera (20) and align the images of all cameras into a single integrated three-dimensional world coordinate system.

[0046] 1) First camera (21): Installed at a specific location on the vehicle, has a specific viewing angle, and captures the surrounding environment to obtain a first image.

[0047] 2) Second camera (22): Installed at a different location from the first camera (21) to capture the surrounding environment and obtain a second image.

[0048] The first camera (21) and the second camera (22) may be positioned so that their viewing angles partially overlap, or there are no overlapping areas.

[0050] 1.3. Pattern pair (30)

[0051] As described above, the present invention utilizes a pair of patterns (30) as a pattern that serves as a reference for camera calibration.

[0052] The above pattern pair (30) includes a first pattern (31) and a second pattern (32).

[0053] In a preferred embodiment of the present invention, the first pattern (31) and the second pattern (32) are not simply separate patterns configured individually, but their relative positional relationship (relative pose) in three-dimensional space is defined in advance.

[0054] In a preferred embodiment of the present invention, the first pattern (31) and the second pattern (32) may be equilateral triangle patterns having the same size and shape, but the present invention is not limited thereto.

[0055] In one embodiment, the first pattern (31) and the second pattern (32) can be set to any geometric shape that facilitates detection and three-dimensional pose estimation, such as a circle, a square, a hexagon, a checkerboard, a grid, or a 2D barcode shape like an ArUco marker (Augmented Reality Utility Codes).

[0056] Preferably, the first pattern (31) and the second pattern (32) are configured to accurately extract unique feature points (corners, center points, contours, etc.) of each pattern and thereby stably estimate the three-dimensional posture.

[0057] In one embodiment, the first pattern (31) and the second pattern (32) may be set to have different sizes or shapes. For example, the first pattern (31) may be circular and the second pattern (32) may be square.

[0059] In a preferred embodiment of the present invention, the first pattern (31) and the second pattern (32) are preferably produced in colors having sufficient contrast (e.g., black and white, high-contrast color combinations) so that they can be clearly detected by a camera.

[0060] In addition, it is preferable to use a matte material for the surfaces of the first pattern (31) and the second pattern (32) to minimize camera flare or specular reflection so that uniform and stable image acquisition is possible even under various lighting conditions, thereby increasing the stability of pattern detection.

[0062] Meanwhile, the geometric relationships between the first pattern (31) and the second pattern (32) constituting the pattern pair (30), such as relative distance (translation), direction (orientation), and rotation in three-dimensional space, are pre-set and pre-stored in the internal database (not shown) of the calibration device (40).

[0063] For example, the geometric center point of the first pattern (31) is pre-set to be separated from the geometric center point of the second pattern (32) by a specific fixed three-dimensional displacement vector (Δx, Δy, Δz) or tilted by a specific rotation matrix or Euler angles (Rx, Ry, Rz), and such information may be stored in an internal database.

[0064] Preferably, information regarding the geometric relationship between the first pattern (31) and the second pattern (32) is obtained from the manufacturing stage of the pattern pair using high-precision measuring equipment (e.g., a three-dimensional coordinate measuring machine (CMM), a laser tracker, a precision robot arm) and can be managed in the form of a database.

[0066] 1.4. Calibration device (40)

[0067] The above calibration device (40) may be a device (hardware) that processes and analyzes images obtained from cameras (21, 22) installed in the vehicle (10), estimates internal and external parameters of the camera, and performs calibration by integrating coordinate systems, or a device in which a software module that performs such functions is installed.

[0068] The calibration device (40) can generally be configured to enable real-time or near-real-time processing by including a high-performance microprocessor (CPU), large-capacity memory (RAM), non-volatile storage device (SSD), and specific image processing accelerators (GPU, FPGA), etc.

[0070] The above calibration device (40) includes the following components.

[0072] 1) Video input unit (41)

[0073] The above video input unit (41) receives the first video and the second video in a real-time streaming or stored form from the first camera (21) and the second camera (22), respectively. This process can be carried out through a wired (e.g., USB 3.0 / C, Gigabit Ethernet, HDMI, etc.) or wireless (e.g., Wi-Fi 6, 5G, etc.) communication interface established between them.

[0074] The image input to the above image input unit (41) is in the form of digital image data, and can improve the accuracy of pattern detection by undergoing a preprocessing process (noise removal, color correction, lens distortion correction, etc.) as needed.

[0076] 2) Pattern detection unit (42)

[0077] The pattern detection unit (42) plays the role of accurately detecting the pattern area required for calibration in each of the first image and the second image received from the image input unit (41).

[0078] In this process, the pattern detection unit (42) may utilize various known image processing and computer vision algorithms as follows.

[0079] a) Edge detection algorithm: The contours of the pattern are extracted using algorithms such as Canny, Sobel, and Laplacian.

[0080] b) Corner detection algorithms: Algorithms such as Harris, Shi-Tomasi, and FAST are used to identify the corners or feature points of the pattern.

[0081] c) Feature extraction and matching algorithms: Unique feature points within the pattern are extracted using algorithms such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), and ORB (Oriented Fast and Rotated BRIEF), and the shape and ID of the pattern are recognized based on these.

[0082] d) Color and brightness-based segmentation algorithm: Separates the pattern region from the background using a specific color range (e.g., green, blue) or brightness threshold.

[0083] e) Machine learning / deep learning-based object detection algorithms: Deep learning models such as YOLO (You Only Look Once), Faster R-CNN, and SSD (Single Shot MultiBox Detector) can be trained in advance with vast amounts of pattern image data to automatically detect the type and location of patterns.

[0084] From the pattern area detected by the pattern detection unit (42), precise geometric 2D image coordinate information necessary for calibration, such as the center point, corner point, contour, and angle of the pattern, is extracted and used for 3D pose estimation in a subsequent step.

[0086] 3) Pattern verification unit (43)

[0087] The pattern verification unit (43) determines the relationship between the patterns in the first image and the second image detected by the pattern detection unit (42) as follows.

[0089] a) Determination of whether the patterns are identical: The feature point descriptors (e.g., SIFT descriptors, etc.) of the patterns detected in the two images are compared with each other, or the unique ID assigned to the pattern (e.g., the ID of an ArUco marker, etc.) is checked to determine whether the two patterns are the same physical object in three-dimensional space.

[0090] For example, if both the first camera (21) and the second camera (22) simultaneously capture the first pattern (31), the pattern verification unit (43) can determine that these two patterns are the 'same pattern'.

[0091] In a preferred embodiment of the present invention, a unique ID assigned to the pattern of the pattern verification unit (43) (e.g., ID of an ArUco marker, etc.) or a feature point descriptor of the pattern detected in two images (e.g., SIFT descriptor, etc.) may be mutually compared, and when the similarity score of a feature point matching algorithm (e.g., FLANN, Brute-Force Matcher, etc.) is greater than or equal to a specific threshold, it may be determined that the two patterns are the same pattern.

[0093] b) Determination of whether the patterns are different: If it is determined that the patterns detected in the two images are not the same object, the pattern verification unit (43) determines whether they are either the first pattern (31) or the second pattern (32) that constitute the pattern pair (30).

[0094] This can also be achieved through the unique ID or feature point matching of each pattern. For example, if the first camera (21) captures only the first pattern (31) and the second camera (22) captures only the second pattern (32), the pattern verification unit (43) determines that these two patterns are 'different patterns'.

[0096] The determination of whether the aforementioned patterns are identical can be automated based on complex image processing techniques and pre-established knowledge regarding pattern pairs.

[0097] For example, the method follows a process where the feature descriptors of detected patterns are compared; if the similarity score exceeds a specific threshold, they are considered the same pattern, otherwise, their respective pattern IDs are checked to determine if they are members of a pattern pair. The reliability of this process is directly linked to the stability of the entire calibration system.

[0099] 4) Coordinate system integration unit (44)

[0100] The above coordinate system integration unit (44) integrates the three-dimensional spatial coordinate systems of the first camera (21) and the second camera (22) based on the judgment result of the above pattern verification unit (43). Details are described in detail through the detailed steps of the calibration method below.

[0102] 2. Calibration Method

[0103] FIG. 3 is a flowchart showing the detailed flow of a camera calibration method according to the present invention. Each detailed step is organically executed by each component of the calibration device (40).

[0105] 2.1. Pattern pair configuration and definition step (S100)

[0106] This step is a preparatory step that must be performed before the calibration process begins. As previously described, in the process of creating a pattern pair (30) including the first pattern (31) and the second pattern (32) of a preferred embodiment of the present invention, a relative positional relationship in three-dimensional space between the two patterns (e.g., relative three-dimensional transformation matrix, rotation and translation vector, etc.) is defined.

[0107] For example, the definition of the geometric relationship between the first pattern (31) and the second pattern (32) can be performed using known professional three-dimensional measuring equipment (CMM, laser scanner, total station, etc.). The data measured through this process can be stored in advance in an internal database or configuration file of the calibration device (40) along with the unique ID of the pattern pair (30).

[0109] 2.2. Step for capturing patterns and acquiring images per camera (S110)

[0110] This step is the starting stage of the process for performing calibration on each camera.

[0111] First, a pair of patterns (30) is placed in front of or around the vehicle (10), or in a specific area where calibration is required.

[0112] In a preferred embodiment of the present invention, the pattern pair (30) may be placed flat on the floor or installed by fixing it to a tripod or a special support. In one embodiment, the pattern pair (30) may be installed on a separate structure provided for installation in a position suitable for calibration work.

[0113] Preferably, the pattern pair (30) should be positioned so that the entire pattern or a part of each pattern falls within the field of view of each camera (21, 22) when installed, and the position of the first pattern (31) and the second pattern (32) constituting the pattern pair (30) can be flexibly determined. This eliminates the constraint of having to secure a common field of view like conventional calibration, thereby greatly increasing the convenience of field work.

[0114] When the installation of the pattern pair (30) is completed as described above, a part or all of the pattern pair (30) is captured through the first camera (21) and the second camera (22) installed in the vehicle to obtain the first image and the second image, respectively.

[0115] The image input unit (41) of the above calibration device (40) receives the first image and the second image in the form of digital data.

[0116] In a preferred embodiment of the present invention, when acquiring the first image and the second image, the calibration device (40) preferably limits the lighting conditions of the image (e.g., uniform lighting, shadow minimization, etc.), the focus of the camera (e.g., sharp image, etc.), the exposure (e.g., prevention of over / under exposure, etc.) to a preset level so that the image of the pattern pair (30) can be clearly identified.

[0118] 2.3. Pattern detection and verification step (S120)

[0119] At this stage, the first image and the second image obtained from the image input unit (41) are processed as follows through the pattern detection unit (42) and the pattern verification unit (43) inside the calibration device (40).

[0120] First, the pattern detection unit (42) performs image processing algorithms such as edge detection, corner detection, feature point extraction, and descriptor generation on each of the input first image and second image to detect image regions within the pattern pair (30).

[0121] In this process, feature points (e.g., corner coordinates, center coordinates, etc.) on the 2D image coordinates of each pattern (31, 32) constituting the pattern pair (30) are extracted to obtain pattern information, and pattern information including 2D coordinates in this way becomes input data for 3D pose estimation.

[0122] The pattern verification unit (43) determines the relationship between the patterns detected in the first image and the second image based on the pattern information detected by the pattern detection unit (42) as described above, using the following two scenarios.

[0124] a) Scenario 1 (same pattern shooting, see FIG. 5): This is a case where the first camera (21) and the second camera (22) have a common shooting area and simultaneously shoot the same pattern, for example, the first pattern (31).

[0125] In this scenario, the pattern verification unit (43) verifies that the patterns detected in the two images are the same physical object in three-dimensional space (e.g., the first pattern (31)). This is achieved through unique ID verification of the pattern, comparison of feature point descriptors, or verification of geometric consistency.

[0127] b) Scenario 2 (shooting different patterns, see FIG. 6): This is a case where the viewing angles of the first camera (21) and the second camera (22) do not overlap at all, or the common shooting area is extremely narrow, so different patterns are shot separately.

[0128] Such situations include cases where there is an immovable obstacle within the shooting range of the first camera (21) and the second camera (22), or cases where part of the first camera (21) and the second camera (22) is obscured by a device or structure of the vehicle on which the first camera (21) and the second camera (22) are installed.

[0129] For example, in this scenario, the first camera (21) may only photograph the first pattern (31) that is not obscured by an obstacle within the shooting range of the first camera (21), and the second camera (22) may only photograph the second pattern (32) that is not obscured by an obstacle within the shooting range of the second camera (22).

[0130] In this scenario, the pattern verification unit (43) verifies that the patterns detected in the first image and the second image are not the same object, and simultaneously identifies that they are the first pattern (31) and the second pattern (32) that constitute a predefined pattern pair (30). This verification is achieved through a matching process using a unique visual identifier (ID or feature point descriptor) of each pattern.

[0132] 2.4. Camera Coordinate System Integration Step (S130)

[0133] This step may proceed differently depending on the judgment result of the pattern verification unit (43) in the aforementioned pattern verification step (S120), that is, depending on the determined scenario, and is performed by the coordinate system integration unit (44) of the calibration device (40).

[0135] In the case of Scenario 1 (same pattern shooting): In this scenario, since the two cameras (21, 22) have captured the same pattern in common, the internal and external parameters of each camera are estimated in a manner similar to the conventional single pattern-based calibration method.

[0137] In the case of Scenario 2 (shooting different patterns): In this scenario, the coordinate system integration unit (44) sets the commonly observed pattern (e.g., the first pattern (31)) as the origin or a specific reference point of any world coordinate system, and each camera (first camera (21), second camera (22)) calculates its own internal parameters and the three-dimensional position and attitude (external parameters) with respect to the world coordinate system using the two-dimensional image coordinates of the detected pattern and the known three-dimensional shape information of the pattern.

[0138] Through this process, the coordinate systems of the first camera (21) and the second camera (22) are established based on the same world coordinate system in the coordinate system integration unit (44), and based on this, the relative transformation relationship between the first camera (21) and the second camera (22) is calculated so that the coordinate systems of the two cameras are integrated.

[0140] In the above scenario 2, the coordinate system integration process of the above coordinate system integration unit (44) is as follows.

[0141] For the sake of convenience of explanation, the following is based on an embodiment in which the first camera (21) captures the first pattern (31) and the second camera (22) captures the second pattern (32), but it should be noted that this is not limited thereto.

[0142] As described above, the relative positional relationship between the first pattern (31) and the second pattern (32) is defined in advance.

[0144] 1) Step (S131) ​​for calculating the relationship between the first camera (C1) and the first pattern (P1)

[0145] The above coordinate system integration unit (44) calculates the relative position and angle relationship between the first camera (C1) and the first pattern (P1) based on the first pattern (P1) detected in the first image.

[0146] For example, the three-dimensional position (a1, b1, c1) and rotation angle of the first camera (C1) can be defined based on the center position of the first pattern (P1). Alternatively, the position and angle of the first pattern (P1) can be defined based on the coordinate system of the first camera (C1).

[0148] 2) Step (S132) to calculate the relationship between the second camera (C2) and the second pattern (P2)

[0149] Likewise, the coordinate system integration unit (44) calculates the relative position and angle relationship between the second camera (C2) and the second pattern (P2) based on the second pattern (P2) detected in the second image.

[0150] For example, the three-dimensional position (a2, b2, c2) and rotation angle of the second camera (C2) can be defined based on the center position of the second pattern (P2).

[0152] 3) Coordinate estimation step using the interrelationship of pattern pairs (S133)

[0153] The calibration device (40) has a predefined relative positional relationship in three-dimensional space between the first pattern (P1) and the second pattern (P2). For example, it is defined that the center position of the second pattern (P2) is separated by (px2, py2, pz2) from the center position of the first pattern (P1) and has a specific rotational relationship.

[0155] 4) Step of displaying all elements on a single coordinate system (S134)

[0156] The calibration device (40) connects the calculated relationships to establish the first pattern (P1), second pattern (P2), first camera (C1), and second camera (C2) on a common coordinate system.

[0157] For example, if the center of the first pattern (P1) is set to the origin (0, 0, 0) of an arbitrary world coordinate system or a specific reference point, the three-dimensional position and orientation of each element can be derived as follows.

[0159] 5) Step for calculating the position and attitude of the first camera (C1) (S135)

[0160] The calibration device (40) calculates the position and orientation of the first camera (C1) in the world coordinate system using the relative three-dimensional positional relationship between the first pattern (P1) and the first camera (C1).

[0162] 6) Step for calculating the position and attitude of the second pattern (P2) (S136)

[0163] The calibration device (40) calculates the position and orientation of the second pattern (P2) in the world coordinate system using a predefined three-dimensional relative positional relationship between the first pattern (P1) and the second pattern (P2).

[0165] 7) Step for calculating the position and attitude of the second camera (C2) (S137)

[0166] The calibration device (40) calculates the position and orientation of the second camera (C2) in the world coordinate system using the position and orientation of the second pattern (P2) calculated through the steps above and the relative three-dimensional positional relationship between the second pattern (P2) and the second camera (C2).

[0168] Through this process, even if the two cameras (C1, C2) capture different patterns (P1, P2), the position and orientation of all cameras and patterns can be established within a single common coordinate system by utilizing the accurate three-dimensional relationship of the pattern pair (30).

[0169] The present invention calculates the relative position and orientation relationship between the first camera and the second camera through the steps described above, and can complete the calibration of the entire system by integrating the coordinate systems of the two cameras.

[0170] The above steps of the present invention enable the effective resolution of multi-camera system calibration problems for irregular vehicles, which were difficult to solve with conventional calibration methods, and are particularly useful in environments where camera installation positions are fluid or field of view is restricted, such as in heavy equipment. Explanation of the symbols

[0174] 10: Vehicle 20: Camera 21: First camera 22: Second camera 30: Pattern pairs 31: Pattern 1 32: Pattern 2 40: Calibration device 41: Video input section 42: Pattern detection unit 43: Pattern verification section 44: Coordinate System Integration Unit

Claims

Claim 1 A device for calibrating a first camera and a second camera installed in a vehicle using a pattern pair comprising a first pattern and a second pattern having a predefined relative positional relationship between them, comprising: a database storing the relative positional relationship in three-dimensional space between the first pattern (P1) and the second pattern (P2) constituting the pattern pair; an image input unit receiving, respectively, a first image and a second image capturing part or all of the pattern pair from the first camera and the second camera; a pattern detection unit detecting a pattern region in each of the first image and the second image; and a pattern verification unit determining whether the pattern detected in the first image and the pattern detected in the second image are the same pattern captured by both cameras in common or different patterns constituting the pattern pair. A camera calibration device comprising: a coordinate system integration unit that integrates the coordinate systems of the first camera and the second camera using a predefined relative positional relationship of the pattern pair when the judgment result of the pattern verification unit determines that the detected patterns are different patterns; wherein the coordinate system integration unit calculates the position and orientation of the first camera (C1) in the world coordinate system using the relative positional relationship of the first pattern (P1) captured by the first camera (C1), calculates the position and orientation of the second pattern (P2) in the world coordinate system using the relative positional relationship between the first pattern (P1) and the second pattern (P2) stored in the database, and calculates the position and orientation of the second camera (C2) in the world coordinate system using the calculated position and orientation of the second pattern (P2) and the relative positional relationship of the second camera (C2), thereby integrating the positions and orientations of the first camera, the second camera, and the pattern pair within a single common coordinate system. Claim 2 delete Claim 3 A camera calibration device according to claim 1, wherein the coordinate system integration unit integrates the coordinate systems of the first camera and the second camera based on the same pattern when the judgment result of the pattern verification unit is that the detected patterns are the same pattern. Claim 4 A camera calibration device according to claim 1, wherein the first pattern and the second pattern have different sizes and shapes. Claim 5 A camera calibration device according to claim 1, wherein the pattern detection unit detects a pattern region using at least one image processing algorithm among edge detection, corner detection, feature point extraction and matching, color and brightness-based segmentation, or deep learning-based object detection. Claim 6 In claim 1, the pattern verification unit determines whether the pattern is the same by comparing feature point descriptors of the detected pattern or verifying a unique ID, a camera calibration device. Claim 7 A method for calibrating a first camera and a second camera installed in a vehicle using a pattern pair comprising a first pattern and a second pattern having a predefined relative positional relationship between them, comprising: (a-1) a step of storing, in a database, the relative positional relationship in three-dimensional space between the first pattern (P1) and the second pattern (P2) constituting the pattern pair; (a) a step of acquiring a first image and a second image by capturing part or all of the pattern pair with the first camera and the second camera in an image input unit; (b) a step of detecting a pattern region in each of the acquired first image and second image in a pattern detection unit; (c) a step of determining, in a pattern verification unit, whether the pattern detected in the first image and the pattern detected in the second image are the same pattern captured in common by the two cameras or different patterns constituting the pattern pair; and (d) a step of integrating the coordinate systems of the two cameras by calculating the relative position and orientation relationship of the first camera and the second camera using the predefined relative position relationship of the pattern pairs, when the detected patterns are determined to be different patterns in the coordinate system integration unit, wherein the step (d) further comprises: (d1) a step of calculating the position and orientation of the first camera (C1) in the world coordinate system using the relative position relationship of the first pattern (P1) captured by the first camera (C1); (d2) a step of calculating the position and orientation of the second pattern (P2) in the world coordinate system using the relative position relationship between the first pattern (P1) and the second pattern (P2) stored in the database; and (d3) a step of calculating the position and orientation of the second camera (C2) in the world coordinate system using the calculated position and orientation of the second pattern (P2) and the relative position relationship of the second camera (C2). Claim 8 delete Claim 9 A camera calibration method according to claim 7, further comprising: (c1) a step of integrating the coordinate systems of the first camera and the second camera based on the same pattern when, as a result of the judgment of step (c) above, the detected patterns are determined to be the same pattern. Claim 10 In claim 7, the above step (c) determines whether the pattern is the same by comparing feature point descriptors of the detected pattern or verifying a unique ID, a camera calibration method.

Images