Method for calibrating automotive camera systems
Patent Information
- Application Number
- JP2026054215
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-03-28
- Filing Date
- 2026-03-27
- Publication Date
- 2026-10-13
AI Technical Summary
【0020】 さらなる利点並びに好ましい特徴及び特徴の組合せは、特に、前述の説明及び特許請求の範囲からもたらされる。以下においては、本発明を図面に基づいて、より詳細に説明する。
Smart Images

Figure 2026165130000001_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method for calibrating a camera system for a motor vehicle, wherein the camera system comprises at least one camera and has a target object, the target object having a known arrangement of circular markers with a marker grid arranged uniformly distributed over a plurality of rows and columns, a digital image of the target object is captured using the camera, and a correction value for calibrating the camera is determined using image evaluation on the basis of the captured image and the known arrangement of the markers.
[0002] The present invention further relates to a device for operating a camera system for a motor vehicle, comprising at least one camera, which device has a control unit that is specially configured to carry out the method described above.
[0003] The present invention further relates to a camera system comprising at least one camera and a device of this type.
Background Art
[0004] Prior Art Camera calibration is a typical component in the manufacture and supply of camera systems, particularly for automotive applications, to ensure that reliable and advantageous image evaluation is guaranteed by the camera system during operation. Most computer vision applications require precise calibration of the intrinsic and extrinsic parameters of the intervening camera. Typically, this calibration is performed before each camera actually begins operation; this is known as offline calibration. However, it is also advantageous to recalibrate the camera during operation to counteract the degradation of individualized parameters that occurs over time, and in particular to enable early detection of camera problems. For this purpose, so-called online calibration is performed during operation. During the calibration process, it is common to use the camera to capture a specially configured target object, i.e., one whose placement is known in particular in its design, in which case the target object is identified in the camera image using an evaluation method, particularly an image evaluation method, and then used for determining camera parameters and, if necessary, for calibration of correction values. Known target objects include checkerboard patterns, overcoded markers, or circular markers arranged in a matrix or grid.
[0005] In the use of circular markers, it is known that the center points of individual circles are first captured, and then these center points are decomposed into a histogram to extract the grid structure of the markers. By using the captured grid structure, potentially identical-looking markers can be uniquely identified based on their position within the grid and assigned to estimated marker positions within the grid structure. Subsequently, the extrinsic and intrinsic parameters of the camera can be determined, for example, by comparing the marker positions detected in the image with the estimated marker positions assigned to the grid structure. It is also known that so-called template matching or template comparison can be performed to detect the center points of the markers, but this template comparison is highly dependent on the size of the circular markers in the image and is therefore not scaling-invariant. This makes the capture of the matrix arrangement particularly sensitive to small changes in the scene determined in the image, such as oblique distortion based on relatively long distances from the target object or changes in rotation angle. Furthermore, template comparison and histogram methods require fine-tuning for each individual use case regarding marker size, number of markers, and marker color.
[0006] Camera calibration typically requires numerous manual steps as preparation for calibration, such as the orientation of the target object relative to the camera, particularly regarding position and orientation, and especially regarding possible rotation angles. In known methods, a "proper" distance between the target object and the camera must also be maintained. The time required for manual preparation and adjustment largely depends on the sensitivity of the image recognition method used, because known methods can react with different sensitivities to changes in the scene. In known methods, to ensure successful image evaluation in comparison to optimal orientation and optimal positioning, for example, an angular deviation of less than 1° and a translational deviation of less than 5cm between the target object and the camera are required. Highly specialized evaluation methods with relatively small recognition errors increase both the time required for calibration and preparation, as well as the cost of carrying out the process. [Overview of the project] [Means for solving the problem]
[0007] Disclosure of the invention The present invention, having the features of claim 1, has the advantage of completely omitting template matching and histograms, and moreover, in this case, the target object having a marker grid can be calibrated with high accuracy without depending on the color and size of the markers, the size of the matrix, and / or the distance between the camera and the target object. The method according to the present invention is as follows: Step a): First, a step is taken in which circular marker candidates are found in the image captured by the camera, Step b): Next, for one or more marker candidates in the image, in particular for each of the captured marker candidates, the neighboring markers are determined from each of the obtained marker candidates. Step c): Next, the vertical and horizontal axes of the candidate marker grid are determined based on the determined adjacent markers. Step d): A step in which all captured marker candidates are identified depending on their own position in the marker candidate grid, Step e): In order to determine the correction values for camera calibration, the requested marker candidate grid and identified marker candidates are compared with known marker grids and target object markers. It is characterized by:
[0008] According to a preferred development of the present invention, in order to carry out step a), an image gradient is determined within the image. For this purpose, for each pixel of the image, the gradient to adjacent pixels is determined, thereby generating a gradient image or edge image. For this purpose, in particular, a Sobel filter and a Canny edge detector are used. By determining the image gradient, a favorable basis is provided for evaluating captured marker candidates.
[0009] Preferably, edge pixels are determined depending on the obtained image gradient. These edge pixels are characterized by a high gradient relative to adjacent pixels, particularly with respect to color and / or brightness. Thus, the edge pixels reproduce the contour of the object captured in the image. With respect to the target object, the image gradient and edge pixel capture result in the circular contour of the marker candidate being highlighted from the captured image, independently of the color and / or brightness of the marker candidate of the target object identified in the image. Therefore, further determination of the marker candidate grid is independent of marker color and marker brightness.
[0010] Preferably, edge pixels that are arranged in a closed circular ring, i.e., edge pixels that form annular edge pixels, are selected as marker candidates. Thereafter, all closed annular shapes of edge pixels are determined to be potential markers for the target object. In relation to the present invention, annular edge pixels are defined in particular as a continuum of adjacent edge pixels, in which case two edge pixels are considered adjacent if they are located in their respective Moore neighborhoods. An edge pixel contour is considered annular, or a closed contour, if the first and last pixels in a series of edge pixels are similarly located in their respective Moore neighborhoods. Preferably, all captured annular edge pixels are selected or classified as marker candidates.
[0011] To capture a circle, in particular, the ratio of the perimeter length to the area of each annular portion consisting of edge pixels is determined. A circular contour is a contour that is equal to or equivalent to a circle; that is, a circular contour may be, for example, circular or elliptical. Circularity itself is preferably determined by the ratio of the perimeter length to the area of the identified marker shape. This ratio is maximum for circles and decreases for other shapes that deviate from circles, such as those approximating ellipses. By adjusting the circular reference of the contour, the method preferably favorably takes into account the oblique distortion and / or intrinsic distortion of the marker.
[0012] Preferably, in step c), up to four adjacent markers located in the nearest neighbor of a candidate marker are determined. First, up to four next adjacent markers are determined for each candidate marker, thereby capturing the actual adjacent parts of the candidate markers, independent of their placement within the identifiable grid. In particular, adjacent markers are selected based on a distance criterion, for example, based on the Euclidean distance.
[0013] From the captured adjacent markers, preferably only adjacent markers located on virtual lines oriented perpendicularly to each other and intersecting within the selected marker candidate are selected. For this purpose, in particular, for each adjacent portion of the marker candidate, it is checked whether there are further adjacent portions of the marker candidate where the delta vector between the marker candidate and the adjacent marker is oriented perpendicularly to each other. Additionally, preferably, spacing constraints are imposed so that, preferably, only adjacent markers that match the distance relationships of other adjacent markers can be accepted. This procedure occurs in the vicinity of the resulting matrix-arranged markers. Preferably, lines or delta vectors are used as the vertical and horizontal axes of the marker candidate matrix.
[0014] Next, correction values for calibrating the camera are advantageously obtained or calculated by known image comparison or calibration methods, depending on the obtained marker candidate grid with uniquely identified marker candidates and the known marker grid with known markers of the target object.
[0015] Preferably, only adjacent markers located on at least one line and having the same distance from a candidate marker are selected, as may be considered in further methods. This achieves the creation and consideration of a uniform matrix pattern.
[0016] Preferably, in step c), the vertical and horizontal axes are selected depending on the virtual line or the delta vector described above.
[0017] More preferably, in step c), it is envisaged that the vertical axis and the horizontal axis are determined by calculation of a covariance matrix and determination of its eigenvectors (Principal Component Analysis, PCA).
[0018] The device according to the present invention having the features of claim 11 is characterized in that the control device is specially configured to carry out the method according to the present invention when used as specified. This provides the advantages already mentioned above.
[0019] The camera system according to the present invention having the features of claim 12 is characterized by the device according to the present invention. This provides the advantages already mentioned above.
[0020] Further advantages as well as preferred features and combinations of features result in particular from the preceding description and the claims. In the following, the present invention will be explained in more detail on the basis of the drawings. [Brief Description of the Drawings]
[0021] [Figure 1] It is a diagram showing a motor vehicle with a camera system in an inspection arrangement. [Figure 2A] It is a diagram showing an exemplary target object for an inspection arrangement. [Figure 2B] It is a diagram showing an exemplary target object for an inspection arrangement. [Figure 3A] It is a diagram showing different inspection situations of a target object. [Figure 3B] It is a diagram showing different inspection situations of a target object. [Figure 3C] It is a diagram showing different inspection situations of a target object. [Figure 3D] It is a diagram showing different inspection situations of a target object. [Figure 3E] It is a diagram showing different inspection situations of a target object. [Figure 3F] It is a diagram showing different inspection situations of a target object. [Figure 4]This figure shows method steps to explain a favorable method for calibrating a camera system. [Figure 5] This figure shows method steps to explain a favorable method for calibrating a camera system. [Figure 6] This figure shows method steps to explain a favorable method for calibrating a camera system. [Figure 7] This figure shows method steps to explain a favorable method for calibrating a camera system. [Figure 8A] This figure shows an example of the results obtained using this method. [Figure 8B] This figure shows an example of the results obtained using this method. [Modes for carrying out the invention]
[0022] Figure 1 shows a schematic plan view of a favorable inspection configuration 1 for the calibration of the camera system 2 of a vehicle 3. Here, the vehicle 3 is positioned in space so that the camera 4 of the camera system 2 can capture a target object 5 in space. According to this embodiment, the target object 5 is configured as a specially recorded display sheet on which multiple markers are displayed in a predetermined arrangement and size.
[0023] Figures 2A and 2B show examples of the target object 5 captured by camera 4, and in particular, stored digital images of the target object 5.
[0024] In this embodiment, multiple markers M are provided, and in Figures 2A and 2B, these markers M are formed as circular markers, and only a few of the markers M are labeled with reference numerals in order to maintain visibility within a predetermined marker grid MR on the front of the target object. Each marker M is a circular surface formed on the target object 5, and they are, for example, different in color or in brightness relative to the background of the target object 5. In the embodiment of Figure 2A, these markers are formed as substantially white circular surfaces on a black background, in which case only two markers are formed as black circular surfaces on a common rectangular white background. In the embodiment of Figure 2B, these markers M are formed as black circular surfaces. In both examples, the marker areas are separated from each other by horizontal stripes having the color of the marker surface. In the embodiment of Figure 2A, a marker grid MR1 consisting of 5 × 6 circular markers M is shown or formed. In the embodiment of Figure 2B, a marker grid MR2 consisting of 2 × 6 circular markers M is formed.
[0025] This type of target object is well known in the automotive industry and is already used in manufacturing plants and workshops to calibrate and inspect cameras and camera systems.
[0026] Figures 3A to 3F show images captured by camera 4, which are used for calibrating camera system 2. For this purpose, target objects 5 located within each image are used. In this case, Figures 3A to 3C show application examples using the target object from Figure 2A, and Figures 3D to 3F show application examples using the target object 5 from Figure 2B.
[0027] According to Figures 3A to 3C, the target object 5 is positioned vertically in space and faces the automobile and its camera 4. In Figure 3A, the target object 5 is located within the optical system of the camera 4 at a certain distance from the automobile and without any particular distortion. In Figure 3B, distortion occurs in the image due to the fisheye objective lens of the camera 4, and this distortion causes the target object 5 to appear curved in space. In the embodiment shown in Figure 3C, the target object 5 is rotated and oriented in space.
[0028] In the embodiment shown in Figure 3D, the target object 5 is located on the ground in front of the car 3 within the camera 4's capture area. In this case, the fisheye optical system causes some distortion of the target object 5 in the image. Figure 3E shows the target object 5 in an even closer location to the car 3, which causes even greater distortion of the target object 5 in the image captured by the camera 4. In the embodiment shown in Figure 3F, the target object 5 is also rotated around its vertical axis, which adds to the distortion of the tilted orientation of the target object 5 in the image from the camera 4.
[0029] The camera system 2 or the automobile 3 also has an advantageous control device 6, which, when used as prescribed, is configured to perform a method for calibrating the camera 4 or the camera system 2, as described below. This method is characterized by four key steps, which are described in detail below.
[0030] Step a) Each captured image is examined for the presence of a circular marker candidate. That is, it is examined whether the image contains elements that could potentially represent the marker M of the target object 5.
[0031] Step b) Next, for each captured marker candidate in the image, the nearest adjacent marker is determined. Therefore, for each marker candidate, the markers adjacent to it, i.e., markers representing adjacent marker candidates, are determined.
[0032] Step c) The vertical and horizontal axes of the marker candidate grid for the captured marker candidate and its neighboring markers are determined, depending on the captured marker candidate and its neighboring markers. Based on the positions and orientations of the marker candidates and their neighboring markers, it is determined whether a marker grid showing a regular distribution of marker candidates on the target object is generated.
[0033] In that case, in the subsequent step d), each marker candidate is uniquely identified based on its own position and orientation within the grid.
[0034] Step e) Next, the captured marker candidate grid with the identified marker candidates and the known marker grid with the known markers of the target object 5 are compared with each other, in particular, using image evaluation or a computer vision algorithm. During this image evaluation, the grid structure, and consequently the position and arrangement of the marker candidates, as well as the size of the marker candidates, are compared with those known markers, thereby determining the distortion in the image of the camera 4 and their orientation and position relative to the target object 5.
[0035] Here, it is assumed that the actual position of the target object 5 relative to the automobile 3 is known and has been accurately set and / or measured in the inspection setup 1, for example, before calibration is performed. In particular, the three-dimensional position relative to the camera is measured or known by a marker M on the target object 5 within the inspection setup.
[0036] In step a), to determine the presence of a circular marker candidate MK, in a favorable method, depending on the captured image, the image gradient of the captured image, particularly the image gradient in the region of the target object 5, is first calculated and displayed, for example, using a Sobel filter and a Canny edge detector. This forms at least an edge image of the target object 5, for example, as shown in Figure 4. The gradient causes the circular transition of the marker candidate to stand out as an edge pixel relative to the background, so that only the contour of the marker candidate MK remains in the gradient image.
[0037] Based on this edge image, the control device 6, in the next step, determines a closed annular contour representing one of the marker candidates MK. In this relationship, a contour is understood as a continuum of adjacent edge pixels, in which case two annular edge pixels are understood to be adjacent if they are located in the Moore neighborhood of the other edge pixel. In order for the contour to be identified as a closed annular contour, it is checked whether the first and last pixels in a series of edge pixels are similarly adjacent and located in the Moore neighborhood of the other pixels.
[0038] Finally, each contour is checked to see if it is a circular contour. The circularity of each contour is preferably calculated based on the ratio of the perimeter length to the contour area of the annular contour. For circles, this ratio is maximized, and for other shapes, such as ellipses, the less this ratio is, the further the contour deviates from a circle. In addition, the control device 6 allows for favorable handling of oblique or intrinsic distortion of the marker candidate MK by adjusting the circular reference of the contour.
[0039] Figure 4 shows the annular contours produced by this procedure, each of which represents a candidate marker MK. Preferably, these candidate markers are represented by different colors to facilitate distinction between adjacent markers.
[0040] In the subsequent steps, the control device 6 determines up to four adjacent markers NM for each marker candidate identified in Figure 4, i.e., adjacent marker candidates located adjacent to the finally selected marker candidate MK. For this purpose, preferably, the four adjacent markers NM located in the closest vicinity of marker candidate MK are first determined by a distance measurement method, such as the Euclidean distance. Here, the adjacent markers NM are selected independently of their own placement in the potential marker grid.
[0041] For this purpose, Figure 5 shows a simplified further procedure based on the three selected marker candidates MK1, MK2, and MK3. Starting from these marker candidates MK1, MK2, and MK3, adjacent marker candidate MK or the nearest marker candidate MK is identified as adjacent marker candidate NM, as visualized by arrows originating from each of the marker candidates MK1 to MK3.
[0042] Preferably, for each adjacent marker NM of each marker candidate MK1 to MK3, it is checked whether there are further adjacent markers NM located orthogonally to each other in the delta vector between the marker candidate and the adjacent marker. In other words, it is checked whether the adjacent markers are located on imaginary lines oriented perpendicularly to each other that intersect with the selected marker candidates MK1 to MK3. Only if this criterion is met is this adjacent marker NM selected and considered for further methods.
[0043] Additionally, distance constraints are considered, so that only adjacent markers NM are selected, or are accepted for further methods, if their distance to collinear adjacent markers is approximately the same as that to candidate marker MK. This leads to a grid-like or lattice-like selection of candidate markers and their adjacent markers NM having a regular distribution, as illustrated in Figure 6. In this way, the candidate marker grid MKR is obtained.
[0044] In this section, it should be noted that the adjacent marker NM may, of course, be considered to be the marker candidate MK itself, used to determine each adjacent marker NM in a further manner.
[0045] Based on the neighborhood relationships of the marker candidate grid MKR or the selected marker candidates MK1 to MK3, the principal axes oriented vertically and horizontally are determined below. For example, these axes are determined by calculating a 2x2 covariance matrix for the delta vectors between each marker candidate MK on the determined marker candidate grid MKR and its neighboring markers NM, and then decomposing the matrix into its eigenvectors (Principal component analysis, PCA). Therefore, in the embodiment of target object 5 of the present invention, the marker candidate MKs in the horizontal direction are further apart from each other than in the vertical direction, so the eigenvectors are shown with relatively large eigenvalues in the horizontal direction. The other eigenvectors are then shown corresponding to the vertical direction of the marker candidate grid MKR, as illustrated by the principal axes x and y written in Figure 6.
[0046] The axes x and y are used below to generate a coordinate system within the image for the marker candidate grid MKR. In this embodiment, the top-left marker candidate MK1 is identified as the marker candidate located closest to the origin of the matrix coordinate system. For this purpose, a graph traverse algorithm, such as a depth search algorithm or a width search algorithm, is used in particular to combine the marker candidate MKs with each other and obtain the marker candidate grid MKR. Then, in particular, the marker candidates located on the marker candidate grid MKR are determined as the marker M of the captured target object 5.
[0047] Each candidate marker data MK is assigned one row and one column within the candidate marker grid MKR, thereby uniquely identifying the candidate marker grid MKR. Preferably, for each candidate marker MK, adjacent markers NM are searched for, and their grid positions are determined based on the direction of passage (horizontal or vertical). This procedure is illustrated in Figure 7.
[0048] The results show that a defined marker can be uniquely determined and identified in the image, as shown in Figures 8A and 8B, regardless of the orientation and orientation of the target object 5 in space relative to the vehicle 3 or camera 4. For this purpose, Figure 8A shows an embodiment of Figure 3B having an identified / marked marker, and Figure 8B shows an embodiment of Figure 3F having an identified / marked marker.
[0049] Depending on the markers identified in the marker candidate grid MKR within the image of camera 4, these markers are compared to known markers or known marker grid MR of the target object 5 in order to examine the intrinsic and / or extrinsic parameters of camera 4 and, if necessary, to calibrate them by determining correction values.
[0050] The procedure described is completely scaling-invariant. The advantage here is that the target object 5, which has a circular marker, can be identified with high reliability, regardless of the scale of the marker in the image, that is, in particular, regardless of the distance from the target object 5 to the camera 4, and there is no need to adjust the parameterization of this method.
[0051] Furthermore, this method is robust against oblique distortion because, for example, it does not require extensive shaping of markers that are likely to fit during template matching. As a result, the target object 5 can be identified with high reliability even if it is tilted, rolling, or rotated in another direction. These advantages allow for a high tolerance for deviations of the actual position and orientation of the target object 5 relative to the target position with respect to the automobile 3 in the inspection setup. This saves a lot of time, for example, during normal camera calibration tasks in production companies or automobile factories, and thus leads to rapid and reliable calibration of the camera 4.
[0052] Furthermore, this method is flexible in terms of the definition and configuration of the target object 5. In particular, this method reliably captures any target object having a grid structure consisting of circular markers. Here, it is not important whether the target object 5 is designed to be colored or colorless. As long as there is a minimum contrast between the marker and its background, this method enables reliable evaluation and calibration. Moreover, the number of rows and / or columns of the marker grid MR can be freely selected, as can the distance that individual markers M must maintain from each other. This results in a high degree of flexibility in the configuration of the target object. Therefore, the proposed method can operate with multiple target objects that already exist and are regularly used, which simplifies the implementation of this method and makes it possible to provide corresponding inspection configurations at low cost.
[0053] This method can operate with synthesized real or arbitrary image data from any image data source, and works with monochrome, colored, or other encoded image data. This method enables the handling of individual or multiple target objects within the same image.
[0054] In summary, the core aspects of the advantageous method according to one embodiment are as follows: First, an image is captured by camera 4 in the inspection setup. Subsequently, the image gradient of this image is determined, and a gradient image is generated. Next, edge pixels are calculated based on the image gradient or gradient image.
[0055] Based on this, closed circular ring contours of edge pixels are captured as marker candidates.
[0056] Based on this, the next adjacent markers are determined for each candidate marker.
[0057] Next, the horizontal and vertical axes of the marker grid are calculated from these adjacency relationships, from the candidate marker to the adjacent marker.
[0058] Based on this, each marker candidate is assigned its own row and column within the marker grid for self-identification, and to that extent, it is determined to be a marker for the target object within the marker grid.
[0059] As a result, each marker identified within the image of camera 4 receives a unique identification and position. Depending on these identifications and positions, parameters to be calibrated for camera 4 are determined when comparing them with the estimated image of the subsequent target object 5, and in particular, correction values for camera 4 calibration are calculated. These correction values are used during the subsequent operation of camera system 2 to ensure optimal image evaluation.
Claims
1. A method for calibrating a camera system (2) for an automobile (1), The camera system (2) comprises at least one camera (4) and a target object (5), the target object (5) having a marker grid (MR) with a known arrangement of circular markers (M) distributed in a plurality of rows and columns. Using the camera (4), a digital image of the target object (5) is captured. In a method in which a correction value for calibrating the camera (4) is determined using image evaluation, depending on the captured image and the known arrangement of the marker (M), The above method involves the following steps: a) A step of examining the image for the presence of circular marker candidates (MK), b) A step of determining a number of marker candidates (MK) in the image or a neighboring marker (NM) for each captured marker candidate (MK), c) A step of determining the vertical and horizontal axes (x, y) of the marker candidate grid (MKR) depending on the obtained adjacent markers (NM), d) The step of identifying the captured marker candidate depending on its own position in the marker candidate grid, e) A step of comparing the marker candidate grid (MKR) with the marker grid (MR) of the target object (5) in order to determine a correction value for the calibration of the camera (4), A method characterized by including
2. In order to perform step a), the image gradient is determined within the image. The method according to claim 1.
3. Depending on the calculated image gradient, edge pixels are captured. The method according to claim 2.
4. Select edge pixels arranged in a closed circular ring as marker candidates (MK). The method according to claim 3.
5. The circular shape is determined by the ratio of the perimeter length to the area of each annular edge pixel. The method according to claim 4.
6. In step c), multiple adjacent markers (NM) located in the nearest vicinity of the marker candidate (MK) are determined, in particular up to four adjacent markers (NM). The method according to any one of claims 1 to 5.
7. Of the captured nearest neighbor markers (NM), only those neighbor markers (NM) located on virtual lines oriented perpendicularly to each other and intersecting within the selected marker candidate (MK) are selected. The method according to claim 6.
8. Only adjacent markers (NM) that are located on at least one line and are the same distance from the candidate marker (MK) are selected. The method according to any one of claims 1 to 7.
9. In step c), the vertical axis and the horizontal axis (x, y) are selected depending on the virtual line. The method according to any one of claims 1 to 8.
10. In step c) above, the vertical axis and the horizontal axis (x, y) are obtained by calculating the covariance matrix and determining its eigenvectors. The method according to any one of claims 1 to 9.
11. In a device for operating a camera system (2) for an automobile (1), which is equipped with at least one camera (4), A control device (6) specially configured to carry out the method described in any one of claims 1 to 10 when used as prescribed, A device characterized by the following features.
12. At least one camera (4) and The apparatus according to claim 11, A camera system equipped with this feature.