Method, computer program and device for determining a calibration matrix for a camera
By detecting moving objects and applying row-by-row and column-by-column scaling, the method effectively calibrates cameras, addressing edge distortions and low light challenges, ensuring accurate image quality and lens compatibility.
Patent Information
- Application Number
- DE102022213952
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-12-19
- Publication Date
- 2024-11-14
- Estimated Expiration
- 2042-12-19
AI Technical Summary
Existing camera calibration methods, particularly for wide-angle cameras, fail to completely remove image distortions at the edges and struggle in low light conditions, leading to unstable calibration results and difficulty in locating features with subpixel accuracy.
A method that detects moving objects between image areas with varying levels of distortion, calculates expected and observed distortions, and determines a calibration matrix based on row-by-row and column-by-column scaling of pixel values, iteratively refining the process to minimize camera-related distortion.
The solution provides accurate calibration results, especially for wide-angle cameras, maintaining image quality and reducing distortion at the edges, while allowing operation with different lenses without requiring defined calibration conditions, and operates during normal operation.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[0001] The present invention relates to a method, a computer program with instructions and a device for determining a calibration matrix for a camera.
[0002] Modern vehicles are increasingly being equipped with cameras to detect the surroundings. The data from these cameras is used by vehicle assistance systems, such as lane departure warning systems and traffic sign recognition systems.
[0003] Comparable prior art is shown, for example, in the documents JP 2011-49 733 A and US 2021 / 0 166 428 A1.
[0004] As (semi-)autonomous vehicles advance, the quality of the images provided by cameras is becoming increasingly important. For this reason, camera calibration is often required to compensate for image distortions caused by the camera lenses, for example. Such distortions are particularly pronounced in wide-angle cameras, for example.
[0005] Different techniques are used to calibrate cameras to eliminate distortion. Many approaches use images of a checkerboard pattern to calculate the camera's intrinsic and extrinsic parameters.
[0006] For example, the article by Z. Zhang, "A Flexible New Technique for Camera Calibration," Microsoft Research Technical Report MSR-TR-98-71 (1998), describes a technique for calibrating a camera that simply requires the camera to observe a planar pattern in two or more different orientations. Either the camera or the planar pattern can be moved freely; the motion does not need to be known. Radial lens distortion is modeled. The procedure consists of a closed-form solution followed by a nonlinear refinement based on the maximum likelihood criterion.
[0007] When using a wide-angle camera, using a checkerboard pattern for calibration cannot completely eliminate distortion at the image edges, as such a purely algebraic optimization leads to unstable calibration results. Furthermore, in low-light conditions, feature localization with subpixel accuracy is very difficult to achieve.
[0008] It is an object of the present invention to provide improved solutions for determining a calibration matrix for a camera.
[0009] This object is achieved by a method having the features of claim 1, by a computer program with instructions having the features of claim 6, and by a device having the features of claim 7. Preferred embodiments of the invention are the subject of the dependent claims.
[0010] According to a first aspect of the invention, a method for determining a calibration matrix for a camera comprises the steps: - Detecting at least one object moving between image areas with different degrees of distortion in a sequence of camera images; - Calculating an expected perspective distortion of at least one object; - determining an observed distortion of the at least one object; - calculating a camera-induced distortion of the at least one object from the observed distortion and the expected perspective-induced distortion; and - Calculate a calibration matrix from the camera-induced distortion.
[0011] According to a further aspect of the invention, a computer program comprises instructions which, when executed by a computer, cause the computer to perform the following steps for determining a calibration matrix for a camera: - Detecting at least one object moving between image areas with different degrees of distortion in a sequence of camera images; - Calculating an expected perspective distortion of the at least one object; - determining an observed distortion of the at least one object; - calculating a camera-induced distortion of the at least one object from the observed distortion and the expected perspective-induced distortion; and - Calculate a calibration matrix from the camera-induced distortion.
[0012] The term "computer" should be understood broadly. In particular, it also includes control units, embedded systems, and other processor-based data processing devices.
[0013] The computer program may, for example, be made available for electronic retrieval or stored on a computer-readable storage medium.
[0014] According to a further aspect of the invention, an apparatus for determining a calibration matrix for a camera comprises: - an object detection module configured to detect at least one object moving between image regions with different degrees of distortion in a sequence of camera images; and - a calculation module configured to calculate an expected perspective-related distortion of the at least one object, to determine an observed distortion of the at least one object, to calculate a camera-related distortion of the at least one object from the observed distortion and the expected perspective-related distortion, and to calculate a calibration matrix from the camera-related distortion.
[0015] The inventive solution is based on the idea of evaluating the distortion of objects in the camera images that move between image areas with different levels of distortion, e.g. from an image area in which they are not distorted or less distorted to an image area in which they are more distorted. In doing so, those components that result from a transformation of the objects due to the perspective geometry are subtracted from the observed distortion. The remaining component of the distortion then results from the camera parameters. The inventive solution has the advantage that the calibration also delivers good results, especially for wide-angle cameras. In addition, it is possible to operate the camera with different lenses without having to perform a calibration under defined conditions after changing the lens. Instead, the calibration takes place during normal operation.
[0016] According to one aspect of the invention, object recognition and object classification are performed to detect the at least one object. For a comprehensive determination of the distortion across the entire image area, it is advantageous to recognize and classify as many objects as possible in the images. Classifying the objects simplifies the subsequent process of identifying identical objects in consecutive images and calculating the distortion resulting from the perspective geometry.
[0017] According to one aspect of the invention, the distance and position of the at least one object are determined in successive camera images. The expected perspective-related distortion is then calculated based on the distance and position. The associated expected distortion in the horizontal and vertical directions can easily be calculated from the change in distance and position in successive images. The distance can be determined, for example, by evaluating the size of an object after a prior classification of the object. To implement the solution according to the invention, it is sufficient if the relative distance and position of the at least one object are known, e.g. with respect to a vanishing point. Even better results can be achieved if the absolute distance and position of the at least one object are known.
[0018] According to one aspect of the invention, so-called feature tracking is carried out for the at least one object, i.e. features of the objects are extracted and tracked within the sequence. The features extracted during feature tracking are then used to calculate the expected perspective-induced distortion and to determine the observed distortion. The use of features increases the amount of data available for calibration, since each detected object generally has a number of features that can be tracked. This allows for a significantly finer calculation of the camera-induced distortion. In addition, the features form geometric figures whose non-linear distortions, depending on the respective distance from the vanishing point, are a measure of the camera-induced distortion. These figures can therefore be additionally evaluated.For this purpose, the expected geometry of a figure in a new image can be calculated based on the detected features. This geometry can then be compared with the actually measured geometry of the figure in the new image, which is derived from the features detected in the new image. The camera-induced distortion can be determined from the deviations detected.
[0019] According to one aspect of the invention, vanishing points in the camera images are determined to calculate the expected perspective-induced distortion. The vanishing points are particularly advantageous when using extracted features as a basis for calculating the camera-induced distortion. Since all tracked features must have a trajectory to the vanishing point, a deviation from this trajectory, particularly in the edge regions of the camera images, is caused by lens distortion. To determine the vanishing point of a camera image, a Hough transform can first be applied to the camera image to find straight lines in the image. If necessary, the camera image can be filtered beforehand, e.g., by converting to grayscale or applying unsharp masking, etc.Using a RANSAC algorithm (RANSAC: Random Sample Consensus), an intersection point of the lines found is then determined. This corresponds to the desired vanishing point.
[0020] According to one aspect of the invention, the calibration matrix is based on a row-by-row and column-by-column scaling of the pixel values of the camera images, where the scaling of the pixel values increases from a central area of the camera images towards the edge areas of the camera images. This approach particularly reduces distortion at wide angles, i.e., in particular, distortion at the edges of the images. On the other hand, the visually important features of the images that are closer to the center of the images are retained. In other words, the less important features are scaled with higher values than the remaining features. The result is scaling of the images with virtually no loss of information and with significantly reduced distortion compared to the prior art.Row-wise and column-wise scaling also helps reduce computation time because it does not require performing a large number of mathematical calculations, as is the case with approaches that use a checkerboard pattern.
[0021] According to one aspect of the invention, the calculated calibration matrix is applied to the camera images. The method steps are then repeated iteratively until minimal camera-induced distortion is achieved. In this way, an optimized calibration matrix is iteratively obtained, which can then be saved or output for further use.
[0022] Preferably, a solution according to the invention is used in a means of transportation to determine a calibration matrix for a camera of the means of transportation. The means of transportation can be, for example, a motor vehicle, an aircraft, a rail vehicle, or a watercraft. The solution according to the invention can also be advantageously applied in robotics.
[0023] Further features of the present invention will become apparent from the following description and the appended claims taken in conjunction with the figures. Figure overview Fig. 1 schematically shows a method for determining a calibration matrix for a camera; Fig. 2 schematically shows a first embodiment of an apparatus for determining a calibration matrix for a camera; Fig. 3 schematically shows a second embodiment of an apparatus for determining a calibration matrix for a camera; Fig. Figure 4 shows schematically a means of transport using a solution according to the invention; Fig. 5-7 shows a sequence of camera images; Fig. 8 illustrates the principle of column-wise scaling; and Fig. Figure 9 illustrates the principle of line-by-line scaling. Character description
[0024] To better understand the principles of the present invention, embodiments of the invention are explained in more detail below with reference to the figures. Like reference numerals are used in the figures for like or equivalent elements and are not necessarily described again for each figure. It is understood that the invention is not limited to the illustrated embodiments and that the described features can also be combined or modified without departing from the scope of the invention as defined in the appended claims.
[0025] Fig. 1 schematically shows a method for determining a calibration matrix for a camera. In a first step, at least one object is detected S1, which moves in a sequence of camera images between image areas with varying degrees of distortion. For this purpose, object recognition and object classification can be carried out, for example. Feature tracking can in particular be carried out for the at least one object S2. To determine the calibration matrix, an expected perspective-related distortion of the at least one object is first calculated S3. As a basis for this step, the distance and position of the at least one object in successive camera images can be determined. Furthermore, vanishing points in the camera images can be determined for this purpose. In addition, an observed distortion of the at least one object is determined S4.For calculating S3 the expected perspective-induced distortion and determining S4 the observed distortion, the features extracted during feature tracking can be used, for example. A camera-induced distortion of the at least one object is calculated S5 from the observed distortion and the expected perspective-induced distortion. A calibration matrix can now be calculated S6 from the camera-induced distortion. This can be applied to the camera images S7. The method steps can then be repeated iteratively until a minimal camera-induced distortion is achieved. The calibration matrix is preferably based on a row-by-row and column-by-column scaling of the pixel values of the camera images, where the scaling of the pixel values increases from a central area of the camera images towards the edge areas of the camera images.
[0026] Fig. Figure 2 shows a simplified schematic representation of a first embodiment of a device 10 for determining a calibration matrix for a camera. The device 10 has an input 11 via which a sequence S of camera images B i a camera 31. An object recognition module 12 is configured to detect at least one object O n to detect that is in the sequence S of camera images B i between image areas with varying degrees of distortion. For this purpose, the object recognition module 12 can, for example, perform object recognition and object classification. For the at least one object, feature tracking can be carried out, in particular, by a feature tracker 13. A computing module 14 is configured to calculate an expected perspective-related distortion of the at least one object O n and to calculate an observed distortion of the at least one object O nAs a basis for calculating the expected perspective-related distortion, the computing module 14 can determine the distance and position of the at least one object in successive camera images. Furthermore, vanishing points in the camera images can be determined for this purpose. For calculating the expected perspective-related distortion and determining the observed distortion, the computing module 14 can, for example, use the features extracted by the feature tracker 13. The computing module 14 is also configured to detect a camera-related distortion of the at least one object O nfrom the observed distortion and the expected perspective-induced distortion, and to calculate a calibration matrix KM from the camera-induced distortion. This can be applied to the camera images. The underlying process can then be repeated iteratively until a minimal camera-induced distortion is achieved. The resulting calibration matrix KM can be stored or output for further use via an output 17 of the device 10. The calibration matrix KM is preferably based on a row-by-row and column-by-column scaling of the pixel values of the camera images B. i , where the scaling of the pixel values from a central area of the camera images B i towards the edges of the camera images B i increases.
[0027] The object recognition module 12, the feature tracker 13, and the calculation module 14 can be controlled by a control module 15. Settings of the object recognition module 12, the feature tracker 13, the calculation module 14, or the control module 15 can be changed via a user interface 18. The data generated in the device 10 can be stored in a memory 16 of the device 10 if necessary, for example, for later evaluation or for use by the components of the device 10. The object recognition module 12, the feature tracker 13, the calculation module 14, and the control module 15 can be implemented as dedicated hardware, for example, as integrated circuits. Of course, they can also be partially or completely combined or implemented as software running on a suitable processor, for example, a GPU or a CPU.The input 11 and the output 17 can be implemented as separate interfaces or as a combined interface.
[0028] Fig. 3 shows a simplified schematic representation of a second embodiment of a device 20 for determining a calibration matrix for a camera. The device 20 has a processor 22 and a memory 21. For example, the device 20 is a control unit or an embedded system. Instructions are stored in the memory 21 which, when executed by the processor 22, cause the device 20 to perform the steps according to one of the described methods. The instructions stored in the memory 21 thus embody a program executable by the processor 22 which implements the method according to the invention. The device 20 has an input 23 for receiving information. Data generated by the processor 22 are provided via an output 24. Furthermore, they can be stored in the memory 21. The input 23 and the output 24 can be combined to form a bidirectional interface.
[0029] The processor 22 may include one or more processor units, such as microprocessors, digital signal processors, or combinations thereof.
[0030] The memories 16, 21 of the described devices can have both volatile and non-volatile memory areas and can comprise a wide variety of storage devices and storage media, for example hard disks, optical storage media or semiconductor memories.
[0031] Fig. Figure 4 schematically shows a means of transport 30 in which a solution according to the invention is implemented. In this example, the means of transport 30 is a motor vehicle. The motor vehicle has a camera 31 for monitoring the surroundings. The images B captured by the camera 31 ican be used by various assistance systems 32, one of which is shown as an example. A device 10 according to the invention is configured to determine a calibration matrix KM for the camera 31, in order, for example, to be able to compensate for the image distortions caused by the camera lenses. Another component of the motor vehicle in this example is a data transmission unit 33. By means of the data transmission unit 33, for example, a connection to a backend can be established, for example, in order to obtain updated software for components of the motor vehicle. A memory 34 is provided for storing data. The data exchange between the various components of the motor vehicle takes place via a network 35.
[0032] In the following, further details of a solution according to the invention are to be given with reference to Fig. 5 to Fig. 9 are described.
[0033] Fig. 5 to Fig. 7 show a sequence of camera images B i . In pictures B i are multiple objects O n , each marked by a bounding box. The distortion of the objects caused by the perspective geometry O n , ie the deformation and scaling of the objects O n , is calculated by determining the distance and position of the moving objects O n in several consecutive images B i is evaluated. The total distortion of the objects O n , ie the perspective distortion plus the camera distortion, is calculated from the distortion and displacement of features M n , n of the objects between consecutive images B i . Some features M n , n are exemplary in Fig. 5 to Fig. 7 marked with arrows.
[0034] According to one aspect of the inventive solution, not all of the image data is scaled or normalized. Instead, features M n , n from pictures B i extracted and the pixel values of the feature images are scaled column by column and row by row. Looking at images B i more precisely, distortions are particularly noticeable at larger angles. This means that the larger the angle, the stronger the images B i stretched in both the x-direction (horizontal) and y-direction (vertical).
[0035] Fig. Figure 8 illustrates the principle of column-wise scaling. If an image is represented in the form of a matrix after feature extraction, an n×m matrix is obtained, as shown schematically in Fig. 8 and Fig. 9. The goal is to scale the pixel values both column-wise and row-wise, performing a kind of ratio scaling. First, we will consider column-wise scaling, which is shown in Fig. 8. Due to the increasing distortion with increasing angle, which is Fig. 8 is illustrated by the density of the column hatching, column 1 on the far left and column 24 on the far right require a larger scaling value than columns 11 to 14 in the middle. Column-by-column scaling therefore means that the value of the scaling factor depends on the column being considered. In particular, the value of the scaling factor decreases as the column number approaches the middle column. In other words, the image is compressed from both the left and right toward the center. This ensures that the image is scaled correctly in the x-direction.
[0036] Fig. Figure 9 illustrates the principle of line-by-line scaling. To scale the image in the y-direction, the procedure is analogous to the x-scaling, but this time from the top and bottom towards the center. Due to the increasing distortion with increasing angle, which is Fig. As illustrated in Figure 9 by the density of the hatching of the rows, the top row 1 and the bottom row 12 require a larger scaling value than rows 6 and 7 in the middle. Therefore, row-by-row scaling means that the value of the scaling factor depends on the row being considered. In particular, the value of the scaling factor decreases as the row number approaches the middle row. In other words, the image is compressed from both the top and bottom toward the center. This ensures that the image is scaled correctly in the y-direction. List of reference symbols 10 Device 11 Entrance 12 Object recognition module 13 Feature Tracker 14 Calculation module 15 Control module 16 storage 17 Exit 18 User interface 20 Device 21 storage 22 processors 23 Entrance 24 Exit 30 means of transport 31 Camera 32 Assistance system 33 Data transmission unit 34 storage 35 Network B i Camera image KM calibration matrix M n , n feature O n object S sequence S1 Detecting an object in a sequence of camera images S2 Performing feature tracking S3 Calculating an expected perspective distortion S4 Determining an observed bias S5 Calculating camera-induced distortion S6 Calculating a calibration matrix S7 Applying the calibration matrix to the camera images
Claims
[1] Method for determining a calibration matrix (KM) for a camera (31), comprising the steps: - Detecting (S1) at least one object (O n ), which is in a sequence (S) of camera images (B i ) moves between image areas with different levels of distortion; - Calculating (S3) an expected perspective-related distortion of the at least one object (O n ), where distance and position of the at least one object (O n ) in consecutive camera images (B i ) and the expected perspective distortion is calculated based on the change in distance and position (S3); - determining (S4) an observed distortion of the at least one object (O n ), whereby for the at least one object (O n) feature tracking is carried out (S2) and for calculating (S3) the expected perspective distortion and determining (S4) the observed distortion, the features extracted during feature tracking (M n,n ) be used; - Calculating (S5) a camera-related distortion of the at least one object (O n ) from the observed distortion and the expected perspective-related distortion; and - Calculating (S6) a calibration matrix (KM) from the camera-induced distortion. [2] Method according to claim 1, wherein for the detection (S1) of the at least one object (O n ) object recognition and object classification is carried out. [3] Method according to one of the preceding claims, wherein for calculating (S3) the expected perspective-related distortion, vanishing points in the camera images (B i ) can be determined. [4] Method according to one of the preceding claims, wherein the calibration matrix (KM) is based on a row-wise and column-wise scaling of the pixel values of the camera images (B i ), in which the scaling of the pixel values from a central area of the camera images (B i ) to the edge areas of the camera images (B i ) increases. [5] Method according to one of the preceding claims, wherein the calculated calibration matrix (KM) is applied to the camera images (B i ) is applied (S7) and the process steps are repeated iteratively until a minimum camera-induced distortion is achieved. [6] Computer program with instructions which, when executed by a computer, cause the computer to carry out the steps of a method according to one of claims 1 to 5 for determining a calibration matrix (KM) for a camera (31). [7] Device (10) for determining a calibration matrix (KM) for a camera (31), comprising: - an object recognition module (12) which is configured to detect at least one object (O n ) to detect (S1) that is in a sequence (S) of camera images (B i ) moves between image areas with different levels of distortion; and - a computing module (14) which is configured to calculate an expected perspective-related distortion of the at least one object (O n ) to calculate (S3) whereby distance and position of the at least one object (O n ) in consecutive camera images (B i ) are determined and the expected perspective-related distortion is calculated on the basis of the change in distance and position (S3), an observed distortion of the at least one object (O n ) to determine (S4), wherein for the at least one object (O n) feature tracking is carried out (S2) and for calculating (S3) the expected perspective distortion and determining (S4) the observed distortion, the features extracted during feature tracking (M n,n ) are used to prevent a camera-related distortion of the at least one object (O n ) from the observed distortion and the expected perspective-induced distortion (S5), and to calculate a calibration matrix (KM) from the camera-induced distortion (S6).
Citation Information
Patent Citations
Camera calibration device and video distortion correction device
JP2011049733A
Sensor self-calibration in the wild by coupling object detection and analysis-by-synthesis
US20210166428A1
JP002011049733A
