A method for extrinsic parameter calibration of vehicle surround view cameras based on depth homography estimation

Abstract

This invention discloses a vehicle surround-view camera extrinsic parameter calibration method based on deep homography estimation. It obtains ground truth labels for homography transformations between adjacent cameras with the ground as the reference surface through a feature matching algorithm. Then, a deep learning model is trained to estimate the homography transformations between adjacent camera images. The estimated homography transformations are then decomposed into pose transformations between adjacent cameras and ground normal vectors. A nonlinear optimization algorithm unifies the translation scales between adjacent cameras. Finally, based on the estimated poses and ground information, the images from each camera are projected onto the ground to obtain a seamlessly stitched 360° top-down view. This invention uses only natural scene images acquired by surround-view cameras for camera-to-ground extrinsic parameter calibration, eliminating the reliance on manual features such as calibration boards and lane lines in traditional calibration. By leveraging a deep learning model, it achieves a camera extrinsic parameter calibration method applicable to various vehicle models and road environments, significantly reducing the calibration cost of surround-view image technology.

Description

Technical Field

[0001] This invention relates to the field of vehicle driver assistance technology, and in particular to a method for calibrating the extrinsic parameters of a vehicle surround-view camera based on depth homography estimation. Background Technology

[0002] With the rapid development of intelligent driving technology, vehicle surround view systems have become an important feature of modern automobiles. Surround view systems acquire images through multiple fisheye cameras installed around the vehicle and stitch these images together to create a 360° top-down view, providing the driver with a panoramic view of the vehicle's surroundings. The key to achieving high-quality image stitching lies in accurately calibrating the extrinsic parameters of each camera, namely the rotation and translation matrices of the camera relative to the ground coordinate system.

[0003] Traditional camera extrinsic calibration methods primarily rely on calibration boards or specific artificial features, such as checkerboard patterns or ArUco markers. These methods require placing calibration objects around the vehicle and extracting feature points through image processing techniques to calculate the camera's extrinsic parameters. While these methods offer high accuracy in controlled environments, they have poor applicability in real-world road environments and the calibration process is cumbersome, making it difficult to meet the calibration needs of vehicles under different road conditions. Another approach is to utilize features from natural scenes, such as lane lines and road textures, for camera calibration. These methods do not require additional calibration objects but depend on specific scene features and are less robust in environments with missing or rapidly changing features. Furthermore, localization and mapping-based methods, such as Simultaneous Localization and Mapping (SLAM), can recover camera pose without relying on calibration objects or specific scenes, but require pre-built environmental maps and can only perform calibration within the scope of these maps. Recently, some deep learning-based surround-view camera extrinsic calibration algorithms have been proposed, but during the training of deep models, they still require supervision information from artificial calibration objects, making it impossible to achieve truly object-free extrinsic self-calibration.

[0004] To achieve extrinsic parameter calibration without relying on any calibration objects, a promising approach is to estimate the homography transformation between cameras instead of directly estimating the pose transformation between the camera and the ground, thereby indirectly recovering the implicit ground information. However, the texture of the road surface area of ​​interest is weak, making it difficult for traditional feature-matching algorithms to extract correct matching point pairs for calculation. Therefore, the powerful capabilities of deep learning methods in computer vision tasks such as image matching and homography estimation are needed. Due to the significant differences in viewpoint between adjacent cameras, unsupervised homography estimation methods are insufficient, requiring ground truth labels to guide model convergence during training. Without calibration objects, ground truth data is unavailable, necessitating a method to obtain homography ground truth labels from image training data. Finally, a method is needed to recover the camera's ground extrinsic parameters from the homography estimates. In summary, designing a methodological framework that combines deep learning methods with extrinsic parameter geometry optimization to achieve a calibration method that utilizes only natural scene images and adapts to various vehicle types and road environments without the need for manual calibration objects is both a challenge and a key focus of the around-view camera extrinsic parameter self-calibration task. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method for extrinsic parameter calibration of vehicle surround-view cameras based on depth homography estimation. Deep learning methods from image matching and homography estimation tasks are introduced into the extrinsic parameter calibration. A deep feature matching algorithm is used to obtain the ground truth homography values ​​between cameras from acquired natural scene images. A depth homography estimation algorithm is then used to estimate the homography transformation between cameras, thereby indirectly recovering the camera's ground-to-ground extrinsic parameters. This method overcomes the limitations of artificial landmarks and specific scenes, is applicable to various vehicle types and road environments, and significantly reduces the calibration cost of surround-view image technology.

[0006] The objective of this invention is achieved through the following technical solution: a method for calibrating extrinsic parameters of a vehicle surround-view camera based on depth homography estimation, comprising the following steps:

[0007] Several surround-view fisheye cameras were used to acquire images around the vehicle. The fisheye images were preprocessed, and adjacent camera image matching pairs were constructed as input image sequences.

[0008] By matching ground feature points of adjacent camera images using an image feature matching algorithm, the true values ​​of homography transformation between adjacent camera images with the ground as the reference plane are obtained through fitting.

[0009] Data augmentation is performed on the input image sequence and the ground value of homography transformation. The pixel-wise correlation between adjacent images is calculated. The ground value of homography transformation is used as supervision. The correlation sampling is input into the deep homography estimation model for training. The trained model is used to obtain the homography estimation value.

[0010] The homography estimate is decomposed to obtain the extrinsic parameters between adjacent cameras and the ground information, and then the extrinsic parameters of the camera relative to the ground are derived.

[0011] Based on the derived extrinsic parameters, the translation scale is optimized by using a nonlinear optimization algorithm that minimizes the reprojection error of the ground matching points, thereby unifying the translation scale between adjacent cameras to the true scale ratio.

[0012] The distortion-free input image is mapped to a top view relative to the ground extrinsic parameters by a camera with a unified translation scale, resulting in a seamlessly stitched 360° top view.

[0013] Furthermore, the preprocessing of the fisheye image specifically includes: firstly, calibrating the intrinsic parameters and distortion coefficients of the panoramic fisheye camera to obtain a fisheye distortion model; then, using the intrinsic parameters and distortion coefficients to perform distortion removal processing on the fisheye image; using the distortion model to split the fisheye image into two distortion-free images with optical axes oriented at 45 degrees to the left and 45 degrees to the right of the original camera's optical axis, respectively; the split images are only used for feature matching and homography estimation, while the image information involved in extrinsic parameter optimization and top-view projection does not include the split images.

[0014] Furthermore, the image feature matching algorithm uses the SuperGlue algorithm to obtain preliminary matching results, normalizes all matching points using camera intrinsic parameters, then uses the RANSAC algorithm to estimate the essential matrix, and removes erroneous matching point pairs; then, it uses the filtered matching point pairs to calculate the homography transformation, uses the RANSAC algorithm to eliminate interference from matching point pairs that are not on the ground, and estimates the homography transformation with the ground as the reference plane.

[0015] Furthermore, the data enhancement specifically includes:

[0016] Data augmentation is performed by adding noise perturbations to the input image sequence and homography ground truth values. The noise takes the form of small-amplitude pose perturbations, including rotation and translation, based on the homography transformation. Relationship with pose:

[0017]

[0018] in Cameras Compared to a camera The rotation matrix and translation vector;

[0019] Distance is obtained from the true value of a single application In the case of a known camera Ground normal vector in coordinate system ,distance In this case, add pose perturbation The corresponding ,use Data augmentation is achieved by transforming the distorted image and the homography truth value respectively.

[0020] Furthermore, the calculation of pixel-wise correlation between adjacent images specifically includes: firstly, extracting high-dimensional features of the image; secondly, performing pixel-wise dot product on the high-dimensional features of adjacent camera images; and thirdly, calculating the pixel-wise correlation between adjacent images for feature maps. Correlation matrix The calculation is as follows:

[0021]

[0022] in, For channel dimension, Images The pixel coordinates, the correlation matrix reflects the image Each location and image The degree of matching between each position.

[0023] Furthermore, the step of using the homography transformation truth value as supervision and inputting the correlation sampling into the deep homography estimation model for training specifically includes:

[0024] Using the previous single-condition estimate For the correlation matrix Sampling is performed to obtain an image. median coordinate Corresponding image Homography transformation coordinates Correlation sampling value This characterizes the evaluation of the accuracy of the current homography estimation; the correlation sampled values ​​are input into the homography transformation estimation network composed of convolutional networks to obtain the homography estimate. The homography estimation network consists of several convolutional layers, normalization layers, activation layers, and pooling layers. It takes correlation sampling values ​​as input and outputs the coordinate offset increments of the four vertices of the image. Since iterative estimation is used, the estimated increment is added to the previous offset to obtain the current vertex offset, i.e. Then, the homography transformation is calculated using the eight-point method. During training, the mean absolute error loss function is used:

[0025]

[0026] in can be Export, Match the set of indices for the camera;

[0027] After obtaining the estimated homography transformation, the homography transformation between the split image and the original distorted image is finally derived from the split image using the known transformation between them.

[0028] Furthermore, the derivation of the camera's extrinsic parameters relative to the ground specifically includes:

[0029] The homography estimate is decomposed to obtain the rotation matrix between cameras, the translation vector, and the camera... Ground normal vector in coordinate system ,distance ;

[0030] By using the constraint that the depth of the matching point is positive, the correct decomposition result is preserved. Starting from the optical center in the first camera coordinate system, along the ground normal vector... direction through distance The projection of the optical center of the first camera onto the ground is obtained and selected as the origin of the ground. The X-axis of the first camera is projected onto the ground as the X-axis of the ground coordinate system, and the ground normal vector is used as the Z-axis of the ground coordinate system. The Y-axis is calculated using the outer product, thus establishing a ground coordinate system and obtaining the transformation from the first camera to the ground coordinate system:

[0031] in, Given the basis vectors of the ground coordinate system, calculate the extrinsic parameters of other cameras relative to the ground coordinate system based on the extrinsic parameters between cameras.

[0032] Furthermore, the process of unifying the translation scale between adjacent cameras to the true scale ratio specifically includes: firstly, obtaining the pixel matching points of adjacent cameras based on the homography estimation value, and projecting the pixel matching points onto the ground using the ground extrinsic parameters; then, minimizing the reprojection error of the ground matching points using the LM nonlinear optimization algorithm, deriving the camera's ground extrinsic parameters from the coordinates of the ground matching points, and further deriving the translation vector between cameras, thereby optimizing the scale of the translation vector, and finally unifying the translation scale between adjacent cameras to the true scale ratio.

[0033] According to another aspect of the specification, a vehicle surround view camera extrinsic parameter calibration device based on depth homography estimation is also provided, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the vehicle surround view camera extrinsic parameter calibration method based on depth homography estimation.

[0034] According to another aspect of the specification, a computer-readable storage medium is also provided, on which a program is stored, which, when executed by a processor, implements the aforementioned method for extrinsic parameter calibration of a vehicle surround-view camera based on depth homography estimation.

[0035] The beneficial effects of this invention: Addressing the problems of traditional surround-view camera extrinsic parameter calibration methods that rely on manual features and have poor environmental adaptability, this invention proposes a calibration method based on deep homography estimation, enabling extrinsic parameter calibration using natural scene images. For common low-texture road surface environments in natural scene images, traditional algorithms struggle to achieve image registration through feature matching. However, the deep learning homography estimation model used in this invention has strong generalization capabilities, achieving accurate estimation results even for low-texture image inputs. This invention obtains the ground truth homography transformation through SuperGlue feature matching of multiple images without relying on manual markers, thereby training a deep learning model to estimate the homography transformation, which is then decomposed to obtain camera extrinsic parameters. Through closed-loop optimization and nonlinear scaling unification, a seamlessly stitched 360° top-down view is finally obtained. Compared to existing surround-view camera calibration schemes that require manual marker assistance, known camera installation locations, or offline calibration to obtain initial estimates of camera extrinsic parameters, this invention achieves self-calibration of surround-view camera extrinsic parameters without any external prior information by introducing a deep learning model. It is applicable to various road environments, supports expansion to various vehicle types, and improves the usability and robustness of the surround-view system. Attached Figure Description

[0036] Figure 1 This is a structural diagram of the method of the present invention;

[0037] Figure 2 This is a flowchart of the training process for the depth homography estimation model of the present invention;

[0038] Figure 3 This is a flowchart illustrating the process of calibrating the extrinsic parameters of a surround-view camera and generating a top view based on depth homography estimation, as described in this invention.

[0039] Figure 4 This is a schematic diagram illustrating the splitting of a fisheye image into two distortion-free images (left and right) in this invention.

[0040] Figure 5 This is a diagram showing the effect of SuperGlue feature matching in this invention;

[0041] Figure 6 This is a top view of the invention, stitched together.

[0042] Figure 7 This is a schematic diagram of the apparatus corresponding to the method of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] This invention aims to address the problems of existing panoramic camera extrinsic parameter calibration methods, which rely on manual features and have poor environmental adaptability. A deep learning-based homography estimation method can robustly estimate the transformation relationships between cameras from natural scene images, providing a new solution for extrinsic parameter calibration. Through homography transformation decomposition, the extrinsic parameters of the camera relative to the ground can be obtained, thereby generating a top-down view. However, the accuracy of homography estimation and the consistency of the translation scale of the extrinsic parameters are key challenges. This invention provides ground truth supervision through SuperGlue feature matching, trains a deep homography estimation model, and unifies the translation scale through nonlinear optimization, ultimately achieving high-precision extrinsic parameter calibration and top-down view stitching.

[0045] like Figure 1 As shown, the present invention provides a method for calibrating the extrinsic parameters of a vehicle surround-view camera based on depth homography estimation, comprising the following steps:

[0046] S1. Use surround-view fisheye cameras installed around the vehicle to acquire images, perform distortion correction processing, and construct adjacent camera image matching pairs as input image sequences;

[0047] S2. For the input image sequence, the ground feature points of adjacent camera images are matched using the SuperGlue algorithm to obtain the true value of homography transformation with the ground as the reference plane.

[0048] S3. Using the homography ground truth as supervision, input the enhanced image and the ground truth, train the deep homography estimation model, and obtain the homography estimate;

[0049] S4. Decompose the homography estimate to obtain the extrinsic parameters between adjacent cameras and the ground information, and then deduce the extrinsic parameters of the camera relative to the ground;

[0050] S5. For the translation vector between adjacent cameras, the pixel matching points are mapped to the ground using the camera's ground extrinsic parameters. A nonlinear optimization algorithm that minimizes the reprojection error of the ground matching points is used to optimize the translation scale, unifying the translation scale between adjacent cameras to the true scale ratio. This yields the camera's ground extrinsic parameters with a determined scale.

[0051] S6. Using the extrinsic parameters after uniform translation scaling, project the distortion-free input image onto the ground to generate a seamless 360° top view.

[0052] The following is a detailed explanation of each step.

[0053] Step S1: Use the surround-view fisheye cameras installed around the vehicle to acquire images, perform distortion correction processing, and construct image matching pairs between adjacent cameras.

[0054] Specifically, it mainly includes the following steps:

[0055] S11. For a panoramic fisheye camera, first calibrate the intrinsic parameters and distortion coefficients. Intrinsic parameters include focal length, principal point coordinates, etc. Distortion coefficients here only consider the radial distortion coefficient of the fisheye camera. The fisheye distortion model is as follows:

[0056]

[0057] in . For normalized coordinates, This represents the angle between the straight line connecting the coordinate point to the camera's optical center and the camera's optical axis, i.e., the radial angle. arrive The change is the radial distortion process. is the radial distortion coefficient.

[0058] S12. For the acquired panoramic fisheye image, distortion correction is performed using intrinsic parameters and distortion coefficients. The coordinate transformation relationship before and after distortion is as follows:

[0059]

[0060]

[0061] in These are the normalized coordinates after distortion. Based on the distortion model and coordinate transformation relationship, the fisheye image can be distorted using calibrated intrinsic parameters and distortion coefficients. To increase the common viewing area between adjacent camera images after distortion correction, facilitating feature matching and homography estimation, this invention uses the distortion model to split the fisheye image into two undistorted images, such as... Figure 4 As shown, its optical axis is oriented 45 degrees to the left and 45 degrees to the right of the original camera's optical axis, respectively, and the horizontal field of view of each image is about 90 degrees, preserving the edge information of the fisheye image to the greatest extent.

[0062] S13. Construct image matching pairs between adjacent cameras. A surround-view system typically includes four cameras: front, right, rear, and left, numbered 1-4. The distortion-corrected split image of camera 1 is denoted as... These are the left and right images, respectively, and so on. To estimate the transformation between adjacent cameras, the input image sequence is constructed as four sets of split image matching pairs: For short: The input sequence composed of the constructed split images is only used for feature matching and homography estimation. Since there is a definite homography transformation between the split images and the original distorted images, it is easy to extrapolate the homography transformation between the split images back to the original images. The image information involved in subsequent extrinsic parameter optimization and top-view projection does not include the split images.

[0063] Step S2: For the input image sequence, the ground feature points of adjacent camera images are matched using the SuperGlue algorithm to obtain the true value of the homography transformation with the ground as the reference plane.

[0064] Specifically, it mainly includes the following steps:

[0065] S21. For the input image sequence, define the approximate pixel range of the ground region in each camera image. Since the panoramic camera's installation position is fixed, the ground region is usually located in the lower part of the image. Based on the panoramic image dataset used, after scaling the input image resolution to a height of 640, the lower half of the image with a ordinate pixel value greater than 150 is taken as the defined ground region. Subsequently, only matching points within this ground region are retained. Actual testing shows that the ground region defined by this method achieves good ground point matching results.

[0066] S22. Use the SuperGlue feature matching algorithm to match feature points between adjacent camera images in the ground region. For adjacent camera image pairs... SuperGlue outputs a set of matching point pairs. , Images The two-dimensional pixels in the image. Because the ground texture is simple, the number of matching points per image pair is too small. Therefore, this invention accumulates 200 pairs of matching points from the images for subsequent homography calculations, with the following effect: Figure 5 As shown. Since the pose and extrinsic parameters of each camera can be considered constant over a short period of time, the error between the homography transformations calculated from the matching points of multiple images can be ignored and can be regarded as the same homography transformation.

[0067] S23. For the initial matching results obtained from SuperGlue, firstly, all matching points are normalized using camera intrinsics. Then, the essential matrix is ​​estimated using the RANSAC algorithm to remove erroneous matching point pairs. Next, the homography transformation is calculated using the filtered matching point pairs, again employing the RANSAC algorithm to eliminate interference from matching point pairs not on the ground and to estimate the homography transformation with the ground as the reference plane. The homography transformation H is a 3x3 matrix describing the transformation between pixel coordinates, satisfying:

[0068]

[0069] in Images The pixel coordinates.

[0070] S24. Multiply the obtained homography transformations clockwise by one revolution to obtain the closed-loop homography product error. For a four-camera system, the closed-loop homography product is:

[0071]

[0072] in, Indicates a front-right single response. This indicates a right-to-back single response, and so on. Ideally, It should be an identity matrix. This can be achieved by minimizing... Nonlinear optimization is used to further optimize each homography transformation, resulting in more consistent homography truth values. .

[0073] Step S3: Using the homography ground truth as supervision, input the enhanced image and the ground truth, train the deep homography estimation model, and obtain the homography estimate.

[0074] Specifically, it mainly includes the following steps:

[0075] S31. Add noise perturbation to the input image sequence and homography ground truth values ​​for data augmentation. The noise takes the form of small-amplitude pose perturbations, including rotation and translation, to simulate the situation where the camera installation position is deviated in real-world application scenarios, thereby improving the model's adaptability to changes in viewpoint. Based on homography transformation... Relationship with pose:

[0076]

[0077] in Cameras Compared to a camera The rotation matrix and translation vector, Cameras Ground normal vector and distance in the coordinate system Cameras Internal reference;

[0078] In the known camera Ground normal vector in coordinate system ,distance In the case of (which can be determined by the above formula) (Decomposed into) pose perturbations The corresponding ,use Data augmentation is achieved by transforming the distorted image and the homography, respectively.

[0079] S32. For the enhanced image, high-dimensional features are extracted using the EfficientNet convolutional network. This invention uses a pre-trained EfficientNet-B0 as the initial weights for the feature extraction network, and the output of the last layer of the network is used as the extracted features. The EfficientNet feature extraction network continuously updates its weights during training.

[0080] S33. Perform pixel-wise dot product on the high-dimensional features of adjacent camera images to calculate the pixel-wise correlation between adjacent images. For feature maps... Correlation matrix The calculation is as follows:

[0081]

[0082] in, The correlation matrix reflects the image's channel dimension. Each location and image The degree of matching between each position.

[0083] S34. Iteratively estimate the homography transformation using feature correlation. Use the previous homography estimate. For the correlation matrix Sampling is performed to obtain an image. median coordinate Corresponding image Homography transformation coordinates Correlation sampling value This characterizes the evaluation of the accuracy of the current homography estimation. The correlation sample values ​​are input into a homography transformation estimation network composed of convolutional networks to obtain the homography estimate. The homography estimation network consists of several convolutional layers, normalization layers, activation layers, and pooling layers. It takes correlation samples as input and outputs the coordinate offset increments of the four vertices of the image. Since iterative estimation is used, the estimated increment is added to the previous offset to obtain the current vertex offset, i.e. Then, the homography transformation is calculated using the eight-point method. During training, the mean absolute error loss function is used:

[0084]

[0085] in can be Export, Match the set of indices for the camera.

[0086] After obtaining the estimated homography transformation, the homography transformation between the split image and the original distorted image is finally derived from the split image using the known transformation between them.

[0087] Step S4: Decompose the homography estimate to obtain the extrinsic parameters between adjacent cameras and the ground information, and then deduce the extrinsic parameters of the camera relative to the ground.

[0088] Specifically, it mainly includes the following steps:

[0089] S41. Homography estimates obtained from decomposition. The relationship between homography transformation and extrinsic parameters and plane information is as follows:

[0090]

[0091] in, Cameras Compared to a camera The rotation matrix and translation vector, Cameras Ground normal vector and distance in the coordinate system Cameras The intrinsic parameters. Using OpenCV's analytical method to obtain the homography matrix. Decompose the data to obtain the external parameters between the cameras. and ground information The decomposition yields four sets of results. The correct decomposition result is retained by constraining the matching point depth to be positive. Due to scale uncertainty, These two distance-related variables cannot determine the scale, and the default decomposition results in... The scale is 1m. In step S5, the scale information between different cameras will be unified.

[0092] S42. In the coordinate system of camera 1 (i.e., the front camera), starting from the optical center of camera 1, along the ground normal vector... direction through distance The projection of the optical center of camera 1 onto the ground is obtained and selected as the origin of the ground coordinate system. Similarly, the X-axis of camera 1 is projected onto the ground as the X-axis of the ground coordinate system, and the ground normal vector is used as the Z-axis of the ground coordinate system. The Y-axis is calculated using the outer product, thus establishing the ground coordinate system. The transformation from the front camera to the ground coordinate system is then:

[0093]

[0094] in, Let be the basis vectors of the ground coordinate system. Based on the extrinsic parameters between cameras, the extrinsic parameters of other cameras relative to the ground coordinate system can be calculated.

[0095]

[0096] Step S5: For the translation vector between adjacent cameras, the pixel matching points are mapped to the ground through the camera's ground extrinsic parameters. The translation scale is optimized and unified to the true scale ratio through a nonlinear optimization algorithm that minimizes the reprojection error of the ground matching points.

[0097] Specifically, it mainly includes the following steps:

[0098] S51. For the problem of scale uncertainty in the translation vector between cameras and the camera-to-ground distance, the matching points of adjacent camera pixels are first obtained based on the homography estimation value. In the image In the lower half of the region, pixels are uniformly sampled at intervals of W / 10 along the horizontal direction in [0, W) and at intervals of H / 10 along the vertical direction in [H / 2, H) to form the image. The set of points to be matched H and W represent the image height and width. The estimated homography transformation is used... , to image Transform the pixel set to the image Obtain the corresponding pixel matching points in the coordinate system Next, the pixel-matching point pairs are projected onto the ground, and the ground in the camera 1 coordinate system is selected as the unified ground representation. For each pixel... First, normalized coordinates are obtained based on intrinsic parameters. Then, the ground projection point is determined along the ray from the optical center to the normalized coordinates. satisfy:

[0099]

[0100] in It's a camera. The k-th ground projection point at the camera Representation in coordinate system It's a camera. Distance to the ground from camera 1 and The directions are opposite, therefore the sign is negative; the above equation constrains... Located on a selected uniform ground surface. Points where the light rays are approximately parallel to the ground are discarded; their corresponding... To avoid excessively distant projection points dominating subsequent optimization, this invention only retains ground projection points with absolute coordinate values ​​less than 10m. The above process is repeated to obtain... Then, using the camera's external parameters calculated in step S4, the ground projection point is obtained. Representation in the terrestrial system:

[0101]

[0102] for The above process is repeated to obtain... .

[0103] S52. Under ideal conditions, the above results and They should perfectly overlap, but due to the scale uncertainty discussed in step S4, during the process of projecting the pixels onto the ground... This is not the correct scale; in other words, the ground in each camera system is not the same ground. Therefore, when projected onto the ground in a unified camera 1 coordinate system... and There is an error between the points. Here, the LM nonlinear optimization algorithm is used to minimize the ground matching points. and To optimize the reprojection error between nodes, thereby reducing the reprojection error. To obtain the correct scale. Specifically, since the ground under camera 1 is selected as the unified ground representation, the ground matching points between cameras 1 and 2 are... Naturally overlap No optimization is needed. Specifically, to incorporate other translations... To achieve complete optimization, a closed-loop link needs to be formed. Therefore, a virtual camera 1' is added at the location of camera 1. The corresponding change And by adding the closed-loop error between camera 1' and camera 1 to the error, the final optimization variable is: .

[0104] definition Then the error vector is:

[0105]

[0106] The reprojection error is defined as:

[0107]

[0108] Optimization is performed using the Levenberg-Marquardt (LM) algorithm, based on the ground matching point coordinates in the error. The camera's external parameters can be derived. This leads to the derivation of the translation vector between cameras. The error vector is thus calculated. For the translation vector between each camera The Jacobian matrix, updated Optimized increments:

[0109]

[0110] in It is a Jacobian matrix. This is the damping factor, controlling the switching between the Gaussian-Newton method and gradient descent. Since only the scale needs to be optimized, it maintains... The direction remains unchanged, and the modulus is adjusted after each iteration. Compared with the optimized update volume Equal. After 10 iterations of optimization, the translation scale between each camera was finally unified to the true scale ratio. During this process, the extrinsic parameters of camera 1 to the ground were... By keeping the parameters constant, the optimized extrinsic parameters of each camera relative to the ground can be calculated based on the optimized inter-camera extrinsic parameters. It is important to note that the optimized translation scale is based on... The initial scale is determined based on the actual scale, and only the scale value after scaling down the actual scale by a certain ratio can be obtained. The true absolute scale cannot be obtained because the pure visual method itself cannot obtain the absolute scale. However, in the 360-degree top-down application scenario of this invention, the lack of absolute scale information only affects the resolution of the final top-down view, which can be adjusted manually and will not affect the stitching quality of the top-down view.

[0111] Step S6: Using the extrinsic parameters after uniform translation scale, project the distortion-free input image onto the ground to generate a seamless 360° top view.

[0112] Specifically, after obtaining the optimized camera-to-ground extrinsic parameters, the origin of the ground coordinate system is further moved from the ground projection point of camera 1 to the center of the ground projection points of the four cameras, so that the center of the subsequent top view approximately coincides with the center of the vehicle. For the projection of each fisheye camera image onto the ground, the mapping from ground coordinates to image coordinates is established as follows:

[0113]

[0114] in, These are the pixel coordinates of the top view. These are the pixel coordinates of the fisheye image. The top-view pixel coordinates are mapped to ground coordinates, then the camera coordinates are obtained through the camera's extrinsic parameters to the ground. The f function represents the distortion process, and finally, the intrinsic parameters are used... Obtain the coordinates of the top view Corresponding fisheye pixel coordinates The top view projection is achieved through bilinear interpolation. For different cameras, corresponding mask regions are set, and only the top view of that region is retained. Finally, the complete top view is obtained by stitching them together, as shown below. Figure 6 As shown.

[0115] This invention introduces a depth homography estimation method into the extrinsic parameter calibration of vehicle surround-view cameras. It provides ground truth supervision through SuperGlue feature matching, leverages the powerful generalization ability of deep learning models to obtain accurate homography transformation estimates, and decomposes these estimates to obtain the camera extrinsic parameters. By unifying the translation scale through nonlinear optimization, the scale uncertainty problem is solved. Ultimately, it achieves extrinsic parameter calibration without manual feature generation and seamless top-down view stitching, improving the usability and robustness of the surround-view system.

[0116] Corresponding to the aforementioned embodiment of a vehicle surround view camera extrinsic parameter calibration method based on depth homography estimation, the present invention also provides an embodiment of a vehicle surround view camera extrinsic parameter calibration device based on depth homography estimation.

[0117] See Figure 7 The present invention provides a vehicle surround view camera extrinsic parameter calibration device based on depth homography estimation, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a vehicle surround view camera extrinsic parameter calibration method based on depth homography estimation in the above embodiment.

[0118] The embodiment of the vehicle surround-view camera extrinsic parameter calibration device based on depth homography estimation provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 7 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is a vehicle surround view camera extrinsic parameter calibration device based on depth homography estimation provided by the present invention. (Except for...) Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0119] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0120] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0121] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements a method for extrinsic parameter calibration of a vehicle surround-view camera based on depth homography estimation as described in the above embodiments.

[0122] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0123] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for extrinsic parameter calibration of a vehicle surround-view camera based on depth homography estimation.

[0124] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0125] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for calibrating extrinsic parameters of a vehicle surround-view camera based on depth homography estimation, characterized in that, Includes the following steps: Several surround-view fisheye cameras were used to acquire images around the vehicle. The fisheye images were preprocessed, and adjacent camera image matching pairs were constructed as input image sequences. The image feature matching algorithm is used to match ground feature points of adjacent camera images and fit the ground to obtain the true value of the homography transformation between adjacent camera images with the ground as the reference plane. The image feature matching algorithm is to use the SuperGlue algorithm to obtain the initial matching result, normalize all matching points using camera intrinsic parameters, estimate the essential matrix using the RANSAC algorithm, and remove erroneous matching point pairs. Then, the homography transformation is calculated using the filtered matching point pairs, and the RANSAC algorithm is used to eliminate the interference of matching point pairs that are not on the ground and estimate the homography transformation with the ground as the reference plane. Data augmentation is performed on the input image sequence and the ground truth of homography. Pixel-wise correlation between adjacent images is calculated. The ground truth of homography is used as supervision. Correlation samples are input into the deep homography estimation model for training. The trained model is then used to obtain homography estimates. Specifically, the step of using the ground truth of homography as supervision and inputting correlation samples into the deep homography estimation model for training includes: The correlation matrix is ​​sampled using the previous homography estimate to obtain the correlation sample value between the coordinates in the image and the corresponding image homography transformation coordinates, which represents the evaluation of the accuracy of the current homography estimation. The correlation sample value is input into the homography transformation estimation network composed of convolutional networks to obtain the homography estimate. The homography estimation network consists of several convolutional layers, normalization layers, activation layers, and pooling layers. The specific form of the output is the estimated coordinate offset increment of the four vertices of the image. Since iterative estimation is used, the estimated increment is added to the previous offset to obtain the current vertex offset. Then, the homography transformation is calculated using the eight-point method. During training, the mean absolute error loss function is used for training. After obtaining the estimated homography transformation, the homography transformation between the original distorted image and the split image is finally derived from the split image using the known transformation between the split image and the original distorted image. The homography estimate is decomposed to obtain the extrinsic parameters between adjacent cameras and the ground information, and then the extrinsic parameters of the camera relative to the ground are derived. Based on the derived extrinsic parameters, a nonlinear optimization algorithm that minimizes the reprojection error of ground matching points is used to optimize the translation scale, unifying the translation scale between adjacent cameras to the true scale ratio. Specifically, unifying the translation scale between adjacent cameras to the true scale ratio includes: first, obtaining the pixel matching points of adjacent cameras based on homography estimation values, and projecting these pixel matching points onto the ground using ground-to-ground extrinsic parameters; then, minimizing the reprojection error of the ground matching points using the LM nonlinear optimization algorithm, deriving the camera's ground-to-ground extrinsic parameters from the ground matching point coordinates, and further deriving the translation vector between cameras, thus optimizing the translation vector scale, and finally unifying the translation scale between adjacent cameras to the true scale ratio. The distortion-free input image is mapped to a top view relative to the ground using camera extrinsic parameters after uniform translation scaling, resulting in a seamlessly stitched 360° top view.

2. The method for calibrating extrinsic parameters of a vehicle surround-view camera based on depth homography estimation according to claim 1, characterized in that, The preprocessing of the fisheye image specifically includes: first, calibrating the intrinsic parameters and distortion coefficients of the panoramic fisheye camera to obtain a fisheye distortion model; then, using the intrinsic parameters and distortion coefficients to perform distortion removal processing on the fisheye image; using the distortion model to split the fisheye image into two distortion-free images with optical axes pointing 45 degrees to the left and right front of the original camera's optical axis, respectively; the split images are only used for feature matching and homography estimation, while the image information involved in extrinsic parameter optimization and top-view projection does not include the split images.

3. The method for calibrating extrinsic parameters of a vehicle surround-view camera based on depth homography estimation according to claim 1, characterized in that, The data enhancement specifically includes: Data augmentation is achieved by adding noise perturbations to the input image sequence and homography ground truth. The noise takes the form of small-amplitude pose perturbations, including rotation and translation. The ground normal vector and distance in the camera coordinate system are obtained based on the relationship between homography transformation and pose. Given the camera coordinate system, the pose perturbation is added to obtain the corresponding homography transformation perturbation. The homography transformation perturbation is used to transform the distorted image and homography ground truth respectively to achieve data augmentation.

4. The method for calibrating extrinsic parameters of a vehicle surround-view camera based on depth homography estimation according to claim 1, characterized in that, The calculation of pixel-wise correlation between adjacent images specifically includes: firstly, extracting the high-dimensional features of the images, performing pixel-wise dot product on the high-dimensional features of adjacent camera images, and calculating the pixel-wise correlation matrix between adjacent images, wherein the correlation matrix represents the degree of matching between each position in the two images.

5. The method for calibrating extrinsic parameters of a vehicle surround-view camera based on depth homography estimation according to claim 1, characterized in that, The derived extrinsic parameters of the camera relative to the ground specifically include: The homography estimate is decomposed to obtain the camera rotation matrix, translation vector, ground normal vector and distance in the camera coordinate system; By using the constraint that the depth of the matching point is positive, the correct decomposition result is preserved. Starting from the optical center in the front camera coordinate system, the projection of the optical center of the front camera onto the ground is obtained along the direction of the ground normal vector through the distance between cameras. This projection is selected as the origin of the ground. The X-axis of the front camera is projected onto the ground as the X-axis of the ground system, and the ground normal vector is used as the Z-axis of the ground system. The Y-axis is calculated using the cross product, thus establishing the ground coordinate system and obtaining the transformation from the front camera to the ground coordinate system. Based on the extrinsic parameters between cameras, the extrinsic parameters of other cameras relative to the ground coordinate system are calculated.

6. A vehicle surround-view camera extrinsic parameter calibration device based on depth homography estimation, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a method for calibrating extrinsic parameters of a vehicle surround-view camera based on depth homography estimation as described in any one of claims 1-5.

7. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a method for calibrating extrinsic parameters of a vehicle surround-view camera based on depth homography estimation as described in any one of claims 1-5.

Images