An automatic driving-oriented multi-fish-eye camera bird's-eye view construction method and system

Abstract

The present application relates to the technical field of vehicle visual perception, and specifically discloses a multi-fish-eye camera bird's-eye view construction method and system for automatic driving, which comprises offline calibration of each fish-eye camera based on a polynomial fish-eye imaging model, construction of a pixel back-projection lookup table, look-up table type distortion removal processing of the collected fish-eye image, extraction of multi-region checkerboard corner points, establishment of a cross-view angle corresponding point set, solution of an initial homography matrix of each fish-eye camera to a bird's-eye reference plane, construction of a joint objective function containing checkerboard re-projection error, overlapping area alignment error and physical scale constraint, global joint optimization of all homography matrices, generation of each road bird's-eye sub-view by using a reverse mapping strategy, and generation of a whole vehicle bird's-eye view based on camera weight difference identification of joint areas. The calibration process of the present application is intuitive, has high geometric accuracy, real-time performance and engineering portability, and is suitable for automatic parking, vehicle surround view systems and automatic driving environment perception.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle visual perception, image geometric projection and embedded real-time computing, and particularly relates to a multi-fish-eye camera bird's eye view construction method and system for automatic driving. BACKGROUND

[0002] The existing vehicle bird's eye view (BEV) system usually relies on four wide-angle or even fish-eye cameras installed in front, back, left and right of the vehicle to generate a top view of the surrounding of the vehicle through projection transformation and image stitching. However, the existing technology generally has the following problems:

[0003] Severe fish-eye distortion, traditional pinhole model cannot meet the accuracy requirement. The field of view angle of the fish-eye lens is usually greater than 180° and the distortion is strong, and it is difficult to establish an accurate imaging relationship by using the pinhole model, which is easy to cause geometric distortion such as bending of straight lines and deformation of objects in the BEV map.

[0004] Multiple cameras lack a unified real scale reference, and the stitching error is significant. The imaging scales of the four directions of the vehicle are often inconsistent, and there is a lack of a unified physical reference, and the stitching area often appears misaligned, stretched or inconsistent in scaling, which affects the accuracy of automatic parking and space measurement.

[0005] The difference between the top view and the ground camera view angle is huge, and the automatic matching is extremely unstable. Due to the difference in perspective distortion, it is difficult to use SIFT / ORB and other general features for automatic matching between the ground camera view and the top view, and the robustness is poor, and it is easy to cause mismatching and registration failure, and the engineering usability is insufficient.

[0006] The calibration process is complicated and non-standard, and it is difficult to be put into engineering. A single checkerboard cannot cover a wide range, and the automatic sorting and cross-view matching of multiple checkerboards are prone to errors, and the calibration process often requires a lot of manual intervention, which has high repeated cost and is difficult to adapt to different vehicle models and camera installation schemes.

[0007] The real-time performance is insufficient, and it is difficult to meet the requirements of vehicle-grade hardware. The calculation amount of fish-eye inverse projection, homography mapping and image fusion is large, and if the complex model is solved online in the running stage and lacks parallel optimization, it is difficult to achieve real-time performance of 30FPS or even higher on the vehicle-grade embedded platform.

[0008] Therefore, there is an urgent need for a multi-fish-eye camera bird's eye view construction method that takes into account high accuracy, robustness, high real-time performance and engineering portability. SUMMARY

[0009] The present application aims to provide a multi-fish-eye camera bird's eye view construction method and system for automatic driving to solve the problems raised in the background.

[0010] To achieve the above object, the present application provides the following technical solutions.

[0011] A multi-fisheye camera bird's eye view construction method for automatic driving, the method comprising:

[0012] Establishing a bird's eye reference plane with unified physical scale constraints;

[0013] Offline calibrating each fisheye camera based on a polynomial fisheye imaging model, constructing a pixel back-projection lookup table, and performing lookup table-based distortion correction on the collected fisheye images;

[0014] Extracting multi-region checkerboard corner points in the perspective-corrected graph and the overhead reference graph, and establishing a cross-view angle corresponding point set;

[0015] Solving the initial homography matrix of each fisheye camera to the bird's eye reference plane based on the corresponding point set, constructing a joint objective function containing checkerboard re-projection error, overlapping area alignment error and physical scale constraint, and globally optimizing all homography matrices;

[0016] Based on the optimized homography matrix, each road bird's eye sub-graph is generated using the reverse mapping strategy;

[0017] Based on the camera weight difference value to identify the joint area, the maximum weight exclusive rendering strategy is used in the non-joint area, and the gamma weighted pixel smoothing fusion is used in the joint area to generate the whole vehicle bird's eye view.

[0018] As a further scheme of the present application, the lookup table-based distortion correction is realized by precomputing the back-projection relationship of the virtual pinhole model to the fisheye image, specifically including:

[0019] A regular pixel grid is constructed with the virtual pinhole model as the target imaging plane;

[0020] According to the polynomial fisheye imaging model, the fisheye image sampling coordinates corresponding to each pixel in the grid are calculated;

[0021] The sampling coordinates are respectively stored as two independent lookup tables map_x and map_y;

[0022] In the running stage, the fisheye image sampling coordinates are directly obtained by lookup table method to realize real-time distortion correction.

[0023] As a further scheme of the present application, the step of extracting multi-region checkerboard corner points in the perspective-corrected graph and the overhead reference graph, and establishing a cross-view angle corresponding point set specifically includes:

[0024] Selecting the region of interest containing the checkerboard in the perspective-corrected graph and the overhead reference graph, respectively;

[0025] The region of interest is zoomed in, and the region of interest after the method is based on the chessboard corner point detection algorithm to obtain the initial position of the corner point and perform sub-pixel optimization;

[0026] The corner points are reordered according to the row and column order of the chessboard, and a one-to-one correspondence relationship of the corner points across the viewing angle is established.

[0027] As a further scheme of the present application, the joint objective function is represented as:

[0028] ;

[0029] Wherein, is the re-projection error of the i-th chessboard corner point of the k-th camera, is the alignment error of the cameras k and l at the boundary of the overlapping region, is the physical scale constraint error, is a set of camera viewing angle pairs that exist in the overlapping region, 、 、 is the weight coefficient of each error.

[0030] As a further scheme of the present application, the inverse mapping strategy specifically includes:

[0031] For each pixel point in the bird's eye reference plane, the floating point sampling position of the pixel point in the perspective corrected image is calculated according to the inverse matrix of the homography of the corresponding camera;

[0032] The pixel value of the floating point sampling position is obtained by using a bilinear interpolation algorithm;

[0033] The calculated pixel value is filled into the corresponding bird's eye sub-image position.

[0034] As a further scheme of the present application, the seam region is identified based on the weight difference value of the camera, and in the non-seam region, a maximum weight exclusive rendering strategy is adopted, and in the seam region, a γ-weighted pixel smoothing fusion step specifically includes:

[0035] For each output pixel position, determine all camera viewing angle sets covering the position;

[0036] Calculate the weight of each camera at the pixel position, determine the maximum weight and the second largest weight, calculate the weight difference between the maximum weight and the second largest weight, and determine the seam region based on the preset threshold of the weight difference value:

[0037] When the weight difference value is greater than the preset threshold, it is determined as a non-seam region, and the pixel of the camera with the maximum weight is used for exclusive rendering;

[0038] When the weight difference value is less than or equal to the preset threshold, it is determined as a seam region, and a γ-weighted linear mixing strategy is used for pixel fusion.

[0039] As a further aspect of the application, the gamma-weighted linear blending strategy comprises:

[0040] Normalizing the camera weights for the pixels in the seam region:

[0041]

[0042] Computing the output pixel value:

[0043]

[0044] wherein, is the maximum weight of the position-dependent weight function, is the set of camera perspectives covering the output pixel, is the pixel value after brightness and color correction.

[0045] As a further aspect of the application, it further comprises an automatic seam path optimization step:

[0046] Constructing a seam cost function in the overlapping region of the at least two aerial sub-views;

[0047] Solving the optimal seam path using a shortest path algorithm or graph cut algorithm with the goal of minimizing the total cost of the seam path;

[0048] Constructing a transition band of a predetermined width on both sides of the optimal seam path;

[0049] Computing a normalized weight for the pixels in the transition band according to their distance to the seam path and fusing them using a gamma-weighted linear blending strategy based on the normalized weights.

[0050] As a further aspect of the application, the seam cost function is expressed as:

[0051]

[0052] wherein: is the corrected pixel value of the selected reference perspective at position x; is the corrected pixel value of the other candidate perspective at position x; is the gradient vector of the reference perspective image at position x, is its gradient magnitude; is a structure saliency mask, taking larger values in structurally salient regions such as car body contour, lane lines, parking lines, etc. and smaller values in background regions; a, b, g are non-negative weight coefficients for balancing the influence of the gradient term, the color difference term and the structure saliency term.

[0053] ​​​The application also provides an automatic driving-oriented multi-fisheye camera bird's eye view construction system, which comprises:

[0054] A reference plane construction module is configured to establish a bird's eye reference plane with unified physical scale constraints;

[0055] A fisheye de-warping module is configured to perform offline calibration on each fisheye camera based on a polynomial fisheye imaging model, construct a pixel back-projection lookup table, and perform lookup table-based de-warping processing on the collected fisheye images;

[0056] A checkerboard extraction module is configured to extract multi-region checkerboard corner points in the de-warping perspective correction image and the overhead reference image, and establish a cross-view angle corresponding point set;

[0057] A homography solving and optimization module is configured to solve initial homography matrices of each fisheye camera to the bird's eye reference plane based on the corresponding point set, construct a joint objective function containing checkerboard re-projection error, overlapping area alignment error and physical scale constraints, and perform global joint optimization on all homography matrices;

[0058] A bird's eye mapping module is configured to generate each road bird's eye sub-image by using a reverse mapping strategy based on the optimized homography matrices;

[0059] A fusion module is configured to identify the seam area based on the camera weight difference value, use a maximum weight exclusive rendering strategy in the non-seam area, and use a gamma-weighted pixel smoothing fusion in the seam area, to generate a whole vehicle bird's eye view.

[0060] Compared with the prior art, the application has the beneficial effects that: the bird's eye view coordinate system with unified physical scale makes the BEVs generated by different cameras and different vehicle models consistent in physical scale, which can be directly used for parking space size verification, parking trajectory planning and space measurement.

[0061] In the running phase, de-warping is completed only by looking up the table, avoiding online solving of complex nonlinear imaging models, greatly reducing the amount of calculation, and supporting real-time processing of multi-channel high-resolution fisheye images on NVIDI A Orin and other embedded platforms.

[0062] Combined with a robust homography solving algorithm with abnormal point rejection, the joint solving of the homography matrices of each view angle improves the numerical stability and noise resistance of the geometric mapping relationship, and reduces the influence of local errors on the overall bird's eye view.

[0063] Global joint optimization is performed on all view angle homography matrices to realize consistent constraints of multi-view geometric mapping relationship, effectively reducing geometric misplacement and scale drift in the splicing area. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application.

[0065] Figure 1 A flow chart of a multi-fisheye camera bird's eye view construction method for autonomous driving is provided for the embodiments of the present application.

[0066] Figure 2 A geometric meaning diagram of joint optimization of multiple camera homography matrices is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0067] In order to make the technical problems to be solved by the present application, technical solutions and beneficial effects more clearly, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0068] Figure 1 A flow chart of a multi-fisheye camera bird's eye view construction method for autonomous driving is provided for the embodiments of the present application.

[0069] A bird's eye view reference plane with unified physical scale constraint is established;

[0070] Each fisheye camera is calibrated offline based on a polynomial fisheye imaging model, a pixel back-projection lookup table is constructed, and a table lookup type distortion correction processing is performed on the collected fisheye image;

[0071] Multi-region checkerboard corner points are extracted in the perspective corrected image after distortion correction and the overhead reference image, and a cross-view angle corresponding point set is established;

[0072] Based on the corresponding point set, an initial homography matrix of each fisheye camera to the bird's eye view reference plane is solved, a joint objective function containing checkerboard re-projection error, overlapping area alignment error and physical scale constraint is constructed, and global joint optimization is performed on all homography matrices;

[0073] Based on the optimized homography matrix, a reverse mapping strategy is used to generate each road bird's eye sub-image;

[0074] Based on the camera weight difference value, the joint area is identified, the maximum weight exclusive rendering strategy is used in the non-joint area, and the γ weighted pixel smoothing fusion is used in the joint area, and the whole vehicle bird's eye view is generated.

[0075] Please refer to Figure 1 and Figure 2In this embodiment, four fisheye cameras are deployed in front of, behind, and on the left and right sides of the vehicle to collect fisheye images covering the perimeter of the vehicle. A drone is used to capture a top-view reference image containing a planar calibration cloth in an attitude where the optical axis is approximately perpendicular to the ground. In the top-view reference image, the scaling factor between the pixels and the actual scale is calculated based on the actual physical size and pixel size of the planar calibration cloth. A bird's-eye view coordinate system with uniform physical scale constraints is established in the bird's-eye view plane, and the top-view reference image is used as the geometric reference of the bird's-eye view plane. This ensures that BEVs generated by different cameras and different vehicle models are consistent in physical scale and can be directly used for parking space size verification, parking trajectory planning, and spatial measurement.

[0076] The scaling factor between pixels and actual scale is obtained by calculating the ratio of the actual physical side length of the checkerboard in the top-view reference image to the pixel length in the image. Based on this scaling factor, the coordinates of the top-view reference image are uniformly mapped to the bird's-eye plane coordinate system with actual physical scale significance, which is used to constrain the scale consistency of the views of each fisheye camera.

[0077] Because fisheye lenses have a large field of view and strong nonlinear distortion, in order to meet the geometric accuracy requirements of bird's-eye projection, this invention uses a fisheye model based on polynomial imaging to calibrate each camera.

[0078] Let θ be the angle between the incident ray and the optical axis, and p be the radius of the corresponding imaging plane. Then the fisheye model can be described as follows:

[0079] ;

[0080] in: These are the model coefficients. Let S denote the distortion center, S be the non-orthogonal correction matrix, and θ be derived from the direction of the ray.

[0081] To ensure that the multi-view fused bird's-eye view is not only geometrically continuous but also possesses a realistic and quantifiable physical scale, this invention utilizes a top-down reference image taken by a drone from a vertical direction as a unified calibration plane. Specifically, this includes:

[0082] With the center of the UAV's top view as the origin, the direction of each pixel is mapped to the x and y axes of a Cartesian coordinate system;

[0083] By identifying the physical dimensions of the ground checkerboard pattern, the scaling factor between pixels and actual scale is calculated: ;

[0084] Establish a BEV output coordinate system consistent with the drone's coordinates on a fixed-resolution bird's-eye view canvas (e.g., 1080×1080);

[0085] When different models of vehicles use the calibration scheme, the mapping relationship of each fisheye camera to the bird's eye plane corresponds to the unified BEV coordinate system described above, thereby ensuring that the bird's eye view generated under different vehicle models and different camera installation schemes has a comparable physical size meaning.

[0086] A chessboard re-projection error term is used to measure the deviation between the re-projection position of the chessboard corner points in the bird's eye plane after being mapped by the current homography matrix and the target position defined by the overhead reference image, so as to constrain the local geometric accuracy in each calibration area.

[0087] An overlapping area boundary alignment error term is used to measure the deviation between the corresponding points obtained by mapping the different cameras in the overlapping area boundary of the bird's eye plane by the respective homography matrices, so as to constrain the splicing consistency between adjacent views.

[0088] A physical scale constraint error term is used to measure the deviation between the distance of the point pairs with known physical distance in the bird's eye plane obtained by the respective homography matrices and the target physical distance calculated by the overhead reference image, so as to ensure that the scale under different angles is consistent with the unified physical scale.

[0089] Referring to Figure 1 and Figure 2 , as a preferred embodiment of the present application, the table lookup distortion removal is realized by precomputing the back-projection relationship of the virtual pinhole model to the fisheye image, specifically including:

[0090] A regular pixel grid is constructed with the virtual pinhole model as the target imaging plane.

[0091] The fisheye image sampling coordinates corresponding to each pixel in the grid are calculated according to the polynomial fisheye imaging model.

[0092] The sampling coordinates are respectively stored as two independent lookup tables map_x and map_y.

[0093] In the running phase, the fisheye image sampling coordinates are directly obtained by table lookup to realize real-time distortion removal.

[0094] In this embodiment, the polynomial imaging model of the fisheye camera is used to calibrate each fisheye camera. In order to meet the real-time requirements of vehicle hardware, the present application does not directly solve the inverse mapping of the above nonlinear imaging model at runtime, but in the offline phase, a regular pixel grid is constructed with the virtual pinhole model as the target imaging plane, and the back-projection relationship of the virtual pinhole model pixels to the fisheye image pixels is calculated according to the polynomial imaging model to generate a pixel back-projection lookup table. The collected fisheye image is subjected to table lookup distortion removal processing to obtain each perspective correction image.

[0095] The table lookup distortion removal is realized by precomputing the virtual pinhole model projection to the fisheye image, and the pixel back-projection lookup table adopts a mapx / mapy pixel mapping table structure, so that the corresponding fisheye image sampling coordinates of each pixel position are quickly obtained by table lookup in the running stage, the complex nonlinear imaging model is avoided to be solved online, the calculation amount is greatly reduced, and the real-time processing of multiple high-resolution fisheye images can be supported on an NVIDIA Orin embedded platform and the like.

[0096] The corresponding sampling coordinates of each pixel in the grid in the original fisheye image are calculated according to the fisheye model, and are recorded as mapx(u,v) and mapy(u,v) respectively, thereby forming a table lookup back-projection mapping structure. The mapx / mapy lookup table structure supports cross-platform loading and can be directly applied to an NVIDIA Orin AGX embedded platform, so that the complete distortion removal of a single fisheye image can be realized within milliseconds.

[0097] Please refer to Figure 1 and Figure 2 As a preferred embodiment of the present application, the step of extracting multi-region checkerboard corner points in the perspective-corrected image and the overhead reference image and establishing a cross-view corresponding point set specifically includes:

[0098] Selecting a region of interest containing a checkerboard in the perspective-corrected image and the overhead reference image respectively;

[0099] Enlarging the region of interest, obtaining initial positions of corner points based on a checkerboard corner point detection algorithm and performing sub-pixel optimization in the enlarged region of interest;

[0100] Reordering the corner points according to the row and column order of the checkerboard to establish a one-to-one correspondence relationship of the cross-view corner points.

[0101] In this embodiment, after the fisheye distortion removal and the bird's-eye plane coordinate system construction are completed, the present application collects multi-region checkerboard feature points in the perspective-corrected image and the overhead reference image of the unmanned aerial vehicle, and establishes a cross-view corresponding relationship. The specific steps are as follows:

[0102] In each perspective-corrected image, a region of interest (ROI) containing a checkerboard is selected by manual or semi-automatic means to reduce background interference; the ROI is appropriately scaled and enlarged to improve the spatial resolution of corner point detection.

[0103] In the enlarged ROI, a checkerboard corner point detection algorithm is used to obtain initial positions of corner points, and the initial corner points are optimized at a sub-pixel level to improve the corner point positioning accuracy.

[0104] According to the chessboard grid row and column order, the corner points of each chessboard grid are manually reordered in row priority or column priority order to ensure that the same physical corner points have consistent index numbers under different viewing angles; the above process is repeated for multiple chessboard grid regions to form a multi-region, cross-view two-dimensional corresponding point set.

[0105] By means of region of interest selection, scaling, sub-pixel corner point extraction and manual reordering, multiple chessboard grid regions are uniformly numbered to form a stable multi-region two-dimensional corresponding point set; combined with a robust homography solving algorithm that excludes abnormal points, the homography matrices of each view are jointly solved to improve the numerical stability and noise resistance of the geometric mapping relationship and reduce the influence of local errors on the overall bird's eye view.

[0106] In the overhead reference image of the unmanned aerial vehicle, the chessboard grid region is also subjected to ROI selection and corner point detection, and a one-to-one correspondence is established with the corner points in the perspective correction image according to the physical position, thereby obtaining a complete matching data set of "overhead reference image coordinates-perspective view coordinates".

[0107] After the chessboard corner point extraction is completed, the multi-view calibration points are mathematically modeled, the robust homography solving algorithm is used to obtain the initial homography matrix of each fisheye camera view to the bird's eye plane, and then a joint objective function is constructed to globally optimize all homography matrices to improve the consistency and accuracy of the multi-view geometric mapping relationship.

[0108] Suppose K fisheye cameras are arranged around the vehicle, and let the i-th corner point coordinate of the k-th camera on the perspective correction image be a two-dimensional column vector:

[0109] ;

[0110] Its corresponding bird's eye view (BEV plane) target coordinate is:

[0111] ;

[0112] where k = 1, 2,..., K, i = 1, 2,..., N k , N k is the number of corner points visible to the k-th camera.

[0113] Introducing homogeneous coordinates, we have:

[0114] ; Let the homography matrix of the k-th camera from the perspective correction image to the bird's eye plane be:

[0115] ; Then under ideal conditions, we have:

[0116] ; wherein, is a non-zero scale factor, so that the current homography matrix The re-projection coordinates of the calculated corner points on the bird's eye plane:

[0117] ;

[0118] Normalized to obtain the two-dimensional re-projection point:

[0119] ; The re-projection error vector of the i-th corner point of the k-th camera is defined as:

[0120] ;

[0121] The corresponding scalar error is:

[0122] ;

[0123] Calculation of the overlap region boundary alignment error term: for any two cameras k and with overlapping regions on the bird's eye plane, define their overlapping region as Select several sampling points in this region, and denote the coordinates of the j-th sampling point on the bird's eye plane as:

[0124] ;

[0125] Using the inverse matrix of the homography matrix , , the bird's eye plane point is back-projected to the corresponding perspective corrected image plane to obtain the homogeneous pixel coordinates:

[0126] ;

[0127] ;

[0128] Normalized to obtain the two-dimensional pixel coordinates:

[0129] ;

[0130] ;

[0131] The overlap region boundary alignment error vector is defined as:

[0132] ;

[0133] The corresponding scalar error is:

[0134] ;

[0135] Physical scale constraint error term calculation: In the calibration process, a number of pairs of ground point pairs with known physical distance can be selected, and the real physical distance of the mth pair of physical points in the bird's eye coordinate system defined by the overhead reference image of the unmanned aerial vehicle is denoted as , m = 1, 2, …, M.

[0136] For the two pixel coordinates corresponding to the point pair on the perspective correction map of the kth camera , , mapped to the bird's eye plane through the homography matrix , we obtain:

[0137] ;

[0138] After normalization, we obtain the two-dimensional coordinates:

[0139] ;

[0140] Thus, we define the pixel space distance derived from the homography matrix:

[0141] ;

[0142] Let ;

[0143] The joint objective function is represented as:

[0144] ;

[0145] wherein is the re-projection error of the i-th checkerboard corner of the kth camera, is the alignment error of the cameras k and l at the boundary of the overlapping region, is the physical scale constraint error, is the set of camera view angle pairs with overlapping regions, , , is the weight coefficient of each error term.

[0146] The independent parameters of each homography matrix are vectorized as optimization variables, and the initial obtained by robust homography solving is used as the initial value. The Levenberg-Marquardt algorithm is used to iteratively solve the joint objective function When the convergence condition is met or the iteration number reaches the preset upper limit, the globally jointly optimized homography matrix and its inverse matrix and write it into a configuration file for calling in a running stage. Through the above joint optimization, the application effectively improves the consistency of geometric mapping between multiple views while ensuring the accuracy of local checkerboard re-projection, and significantly reduces the geometric misalignment of the splicing area.

[0147] Based on a robust homography solving algorithm containing outlier rejection, the initial homography matrix of each fisheye camera view to the bird's eye plane is obtained by using the two-dimensional corresponding point set of each region; on this basis, taking each initial homography matrix as the initial value, a joint objective function containing the re-projection error of the corner points of the checkerboard in the bird's eye plane, the alignment error of different cameras at the boundary of the overlapping area of the bird's eye plane, and the physical scale constraint of the bird's eye plane is constructed, and a nonlinear least squares optimization algorithm is used to iteratively solve the joint objective function, thereby obtaining the homography matrix of each fisheye camera view to the bird's eye plane after global joint optimization.

[0148] The initial homography matrix of each fisheye camera view is obtained by jointly solving the two-dimensional corresponding points in multiple checkerboard regions, and the random sample consensus algorithm is used to reject outliers in the robust homography solving process to improve the numerical stability and noise resistance of the initial homography matrix.

[0149] Please refer to Figure 1 and Figure 2 As a preferred embodiment of the application, the inverse mapping strategy specifically includes:

[0150] For each pixel point in the bird's eye reference plane, the floating-point sampling position of the pixel point in the perspective correction graph is calculated according to the inverse matrix of the homography of the corresponding camera;

[0151] The pixel value of the floating-point sampling position is obtained by using a bilinear interpolation algorithm;

[0152] The calculated pixel value is filled into the corresponding bird's eye subgraph position.

[0153] In this embodiment, during the running of the vehicle, the perspective correction graph is obtained by calling the pixel back-projection lookup table for the real-time collected fisheye image, and the sampling position of each pixel point in the bird's eye plane in the corresponding perspective correction graph is inversely calculated according to the globally jointly optimized homography matrix by using the inverse mapping strategy, and the pixel value is obtained based on the sampling coordinates by using the bilinear interpolation, thereby generating each bird's eye subgraph.

[0154] The above process can be completed on an embedded platform such as NVIDIA Orin AGX with the help of a CUDA acceleration module, thereby realizing parallel projection and synchronous output of multiple cameras in a real-time environment.

[0155] The reverse mapping strategy specifically includes: for each pixel in the bird's-eye view plane, calculating its floating-point sampling position in the perspective-corrected image based on the inverse matrix of the homography matrix of the corresponding camera, and obtaining pixel values ​​for integer pixels near the floating-point sampling position using bilinear interpolation or other interpolation algorithms.

[0156] Please see Figure 1 and Figure 2 As a preferred embodiment of the present invention, the steps of identifying seam regions based on camera weight differences, employing a maximum weight exclusive rendering strategy in non-seam regions, and using γ-weighted pixel smoothing fusion in seam regions specifically include:

[0157] For each output pixel location, determine the set of all camera viewpoints covering that location;

[0158] Calculate the weight of each camera at the pixel position to determine the maximum and second-largest weights; calculate the weight difference between the maximum and second-largest weights, and determine the seam area based on a preset threshold of the weight difference:

[0159] When the weight difference is greater than the preset threshold, it is determined to be a non-seamless area, and the camera pixel with the largest weight is used for exclusive rendering.

[0160] When the weight difference is less than or equal to a preset threshold, it is determined to be a seam area, and a γ-weighted linear mixing strategy is used for pixel fusion.

[0161] In this embodiment, after obtaining the various bird's-eye view sub-images, the pixels in the overlapping areas of multiple viewpoints in the bird's-eye view plane are subjected to seam area recognition and adaptive fusion to reduce the brightness difference and geometric discontinuity of the multi-camera stitching area and improve the visual continuity of the whole vehicle bird's-eye view.

[0162] Let the output bird's-eye view plane be defined as the region. Its arbitrary pixel position is denoted as The pixel value of the bird's-eye view sub-image of the k-th camera at position x is denoted as... The pixel value after brightness and / or color correction is denoted as .

[0163] Brightness and / or color correction in overlapping areas:

[0164] For any pair of cameras k and k with overlapping regions The overlapping area of ​​its bird's-eye view plane is denoted as Within this region, the mean and variance of the brightness (or single-channel color) of the k-th subimage are defined as follows:

[0165] ;

[0166] No. The corresponding statistics for the path diagram are:

[0167] ;

[0168] With the kth subgraph as reference, the kth subgraph is corrected by linear transformation:

[0169] ;

[0170] Where the gain coefficient and the offset are:

[0171] ;

[0172] To prevent the denominator from being zero, a small positive number is added. For color images, the above statistics can be calculated for each channel R, G, B respectively and linear correction is performed independently for each channel. The corrected pixel is denoted as .

[0173] For each output pixel , define the set of camera view angles covering the pixel:

[0174] ;

[0175] For pixels belonging to the multi-view overlapping region (i.e. ), define a position-dependent weight function for each camera in advance, for example, based on the distance from the pixel to the center of the camera field of view, to the center of the vehicle, or to the boundary of the respective field of view. For pixel x, denote the largest and the second largest view angle weight of x as:

[0176] ;

[0177] ;

[0178] Where is the index of the camera with the largest weight .

[0179] Define the weight difference value:

[0180] ;

[0181] Given a preset threshold , define the seam indicator function:

[0182] ;

[0183] Where, indicates that pixel x belongs to the seam region, ​x is a pixel belonging to a non-seam region.

[0184] The preset threshold is used to divide the pixels in the multi-view overlapping region into a seam region and a non-seam region, when the difference between the maximum weight and the second maximum weight is greater than the preset threshold, the corresponding pixel is determined as a non-seam region and the maximum weight is used for exclusive rendering; when the difference is less than or equal to the preset threshold, the corresponding pixel is determined as a seam region and a multi-view fusion strategy is used.

[0185] Please refer to Figure 1 and Figure 2 As a preferred embodiment of the present application, the gamma-weighted linear mixing strategy includes:

[0186] For the pixel x satisfying or , that is, the pixel located in the seam path and the transition zone thereof, a gamma-weighted multi-view smooth fusion strategy is used. First, the camera weights covering the seam region pixels are normalized:

[0187] ;

[0188] Then, an exponential parameter γ>0 is introduced to calculate the output pixel value:

[0189] ;

[0190] Wherein, is the maximum weight of the position-varying weight function, is the camera view set covering the output pixel, is the pixel value after brightness and color correction.

[0191] In the present embodiment, when γ>1, the view with a larger weight further increases the proportion in the fusion result, and in the seam region, since the weights of the views are similar, the gamma-weighted linear mixing can form a smooth transition between the multiple views, thereby obtaining a natural fusion effect of brightness and texture continuity in the seam region.

[0192] After normalizing the weights of the views covering the seam region pixels, the normalized weights are amplified by γ power, and then the pixel values of at least two cameras are linearly weighted according to the amplified weights, so as to realize the smooth transition between the multiple views while ensuring the clarity of the main view.

[0193] Please refer to Figure 1 and Figure 2 As a preferred embodiment of the present application, it further includes an automatic seam path optimization step:

[0194] In the overlapping region of the at least two bird's eye view subgraphs, a seam cost function is constructed;

[0195] The shortest path algorithm or graph cut algorithm is used to solve the optimal seam path with the minimum total cost of the seam path as the target.

[0196] A transition band with a preset width is constructed on both sides of the optimal seam path.

[0197] The normalized weight of a pixel in the transition band is calculated according to the distance of the pixel to the seam path, and a gamma-weighted linear blending strategy based on the normalized weight is used for fusion.

[0198] In the embodiment, in order to make the seam pass around the structure salient regions such as the vehicle body contour, lane line and parking space line as much as possible, a seam cost map is constructed in the multi-view overlapping region. For any pixel , the seam cost function is expressed as:

[0199] ;

[0200] wherein: is the corrected pixel value of the selected reference view at position x; is the corrected pixel value of another candidate view at position x; is the gradient vector of the reference view image at position x, is the gradient amplitude thereof; is the structure saliency mask, which takes a larger value in the structure salient regions such as the vehicle body contour, lane line and parking space line, and takes a smaller value in the background region; and α, β and γ are non-negative weight coefficients for balancing the influences of the gradient term, color difference term and structure saliency term.

[0201] The seam path is expressed as a sequence of connected pixels from one side boundary of the overlapping region to the other side boundary:

[0202] ;

[0203] The total cost of the seam path is defined as:

[0204] ;

[0205] The above objective function is optimized by the shortest path algorithm, dynamic programming algorithm and / or graph cut algorithm to obtain the seam path with the minimum total cost:

[0206] ;

[0207] After the optimal seam path is obtained, a transition band region with a fixed width can be constructed on both sides of the optimal seam path to form a smooth weight transition near the seam.

[0208] Maximum weight exclusive rendering of the non-seam region:

[0209] For pixels x satisfying and , the maximum weight exclusive rendering strategy is adopted.

[0210] ;

[0211] The pixel value of the bird's eye view at position x is output as:

[0212] .

[0213] All calibration and fusion related parameters (including fisheye imaging model parameters, pixel back-projection lookup table, homography matrix and its inverse matrix, camera weight, seam mask, and brightness and / or color correction parameters, etc.) are saved in a configuration file in a unified format, preferably in YAML format. During the initialization stage of system running, the pipeline management module loads each type of parameter in the configuration file in turn, automatically builds a bird's eye view processing pipeline including fisheye de-warping, bird's eye projection and multi-view fusion, without the need to recompile or modify the algorithm logic at the code level.

[0214] When the camera installation position, vehicle width, UAV shooting height or hardware platform changes, only need to re-execute the steps in the calibration environment and get new calibration results, and update the new fisheye imaging model parameters, lookup table back-projection mapping, homography matrix and related fusion parameters to the configuration file, to complete the rapid adaptation to new vehicle models or new hardware platforms. The parameter-driven pipeline design makes the bird's eye view construction method of the present application maintain high geometric accuracy and real-time performance while having good engineering portability and maintenance convenience, significantly reducing the deployment and upgrade cost of the system.

[0215] The present application combines UAV overhead view, fisheye lookup table de-warping, multi-region checkerboard joint calibration, real-time homography mapping and seam-aware fusion to form a complete and engineering deployable BEV construction scheme. This scheme not only improves the geometric accuracy of the bird's eye view, but also has strong real-time performance and adaptability, and is more suitable for engineering deployment of autonomous vehicles.

[0216] On the NVIDIA Orin AGX platform, the present application method can achieve real-time processing frame rate meeting vehicle requirements under the condition of four-way 1920x1536 resolution fisheye image input after combining with CUDA parallel implementation. Compared with the comparative scheme without using lookup table de-warping and joint optimization, the present application significantly reduces the corner reprojection error in the stitching area and significantly reduces the bird's eye view seam artifacts.

[0217] Before determining the seam region by using the camera weight difference value, an automatic seam path optimization step is further included, specifically comprising: constructing a seam cost map in the overlapping region of the at least two bird's eye subgraphs according to pixel gradient intensity, structure and / or semantic target distribution and color difference, assigning a cost value to each pixel position in the overlapping region, so that the cost value of structure significant areas such as vehicle body contour, lane line and parking line is higher than that of background areas; representing the seam path as a connected pixel path from one side boundary to the other side boundary of the overlapping region, taking the total cost value of the path as the optimization target, solving the seam path with the minimum total cost value by using the shortest path algorithm and / or graph cut algorithm, and using the seam path to limit the position distribution of the seam region.

[0218] A transition band with a preset width is constructed along both sides of the seam path, a normalized weight is calculated for a pixel in the transition band according to its distance to the seam path, and a gamma-weighted linear blending strategy based on the normalized weight is used to fuse the pixels of the at least two cameras, so as to obtain output pixels that are smoothly and gradually changed in the seam region, thereby further weakening the brightness mutation and texture discontinuity at the seam.

[0219] In the embodiment of the present application, a multi-fish-eye camera bird's eye view construction system for automatic driving is also provided, and the system comprises:

[0220] A reference plane construction module is configured to establish a bird's eye reference plane with unified physical scale constraints;

[0221] A fish-eye de-warping module is configured to perform offline calibration on each fish-eye camera based on a polynomial fish-eye imaging model, construct a pixel reverse projection lookup table, and perform lookup table-based de-warping processing on the collected fish-eye images;

[0222] A checkerboard extraction module is configured to extract multi-region checkerboard corner points in the de-warping perspective correction graph and the overhead reference graph, and establish a cross-view angle corresponding point set;

[0223] A homography solving and optimization module is configured to solve initial homography matrices of each fish-eye camera to the bird's eye reference plane based on the corresponding point set, construct a joint objective function including checkerboard re-projection error, overlapping region alignment error and physical scale constraint, and perform global joint optimization on all homography matrices;

[0224] A bird's eye mapping module is configured to generate each bird's eye subgraph by using a reverse mapping strategy based on the optimized homography matrices;

[0225] A fusion module is configured to identify the seam region based on the camera weight difference value, use a maximum weight exclusive rendering strategy in the non-seam region, and use a gamma-weighted pixel smoothing fusion in the seam region, to generate an entire vehicle bird's eye view.

[0226] The above merely describes preferred embodiments of the present application, and is not used to limit the present application, any modification, equivalent replacement and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for constructing an aerial view of a multi-fisheye camera for autonomous driving, characterized in that, The method comprises: establishing an aerial view reference plane with a unified physical scale constraint; performing offline calibration on each fisheye camera based on a polynomial fisheye imaging model, constructing a pixel back-projection lookup table, and performing lookup table-based distortion correction on the collected fisheye image; extracting multi-region checkerboard corner points in the perspective-corrected image and the overhead reference image, and establishing a cross-view angle corresponding point set; solving an initial homography matrix of each fisheye camera to the aerial view reference plane based on the corresponding point set, constructing a joint objective function containing checkerboard re-projection error, overlapping area alignment error and physical scale constraint, and performing global joint optimization on all homography matrices; generating each aerial view sub-image using a reverse mapping strategy based on the optimized homography matrix; identifying the seam area based on the camera weight difference value, using the maximum weight exclusive rendering strategy in the non-seam area, and using the gamma-weighted pixel smoothing fusion in the seam area to generate the whole vehicle aerial view.

2. The method of claim 1, wherein, The lookup table-based distortion correction is achieved by precomputing the back-projection relationship from the virtual pinhole model to the fisheye image, and specifically comprises: constructing a regular pixel grid with the virtual pinhole model as the target imaging plane; calculating the fisheye image sampling coordinates corresponding to each pixel in the grid according to the polynomial fisheye imaging model; storing the sampling coordinates as two independent lookup tables, map_x and map_y, respectively; directly obtaining the fisheye image sampling coordinates in the running stage through the lookup table method to realize real-time distortion correction.

3. The method of claim 1, wherein, The step of extracting multi-region checkerboard corner points in the perspective-corrected image and the overhead reference image, and establishing a cross-view angle corresponding point set specifically comprises: selecting the regions of interest containing the checkerboard in the perspective-corrected image and the overhead reference image, respectively; enlarging the regions of interest, and obtaining the initial position of the corner points and performing sub-pixel optimization based on the checkerboard corner point detection algorithm in the method; reordering the corner points according to the row and column order of the checkerboard to establish a one-to-one correspondence between the cross-view angle corner points.

4. The method of claim 1, wherein, The joint objective function is expressed as: ； wherein, is the reprojection error of the i-th corner of the k-th camera, is the alignment error of the cameras k and l at the border of the overlap region, is the physical scale constraint error, is the set of pairs of camera views that have an overlap region, , , are the weight coefficients of the respective errors.

5. The method of claim 1, wherein, The reverse mapping strategy specifically comprises: for each pixel point in the aerial view reference plane, calculating its floating-point sampling position in the perspective-corrected image according to the homography inverse matrix of the corresponding camera; obtaining the pixel value by using the bilinear interpolation algorithm on the floating-point sampling position; filling the calculated pixel value into the corresponding aerial view sub-image position.

6. The method of claim 1, wherein, The step of identifying the seam area based on the camera weight difference value, using the maximum weight exclusive rendering strategy in the non-seam area, and using the gamma-weighted pixel smoothing fusion in the seam area specifically comprises: for each output pixel position, determining the set of all camera views covering the position; calculating the weight of each camera at the pixel position to determine the maximum weight and the second largest weight; calculating the weight difference between the maximum weight and the second largest weight, and determining the seam area based on the preset threshold of the weight difference: when the weight difference is greater than the preset threshold, it is determined as a non-seam area, and the pixel of the camera with the maximum weight is used for exclusive rendering; when the weight difference is less than or equal to the preset threshold, it is determined as a seam area, and a gamma-weighted linear mixing strategy is used for pixel fusion.

7. The method of claim 1, wherein, The gamma-weighted linear mixing strategy comprises: Normalizing the camera weights of the pixels in the overlap region: ； Calculating the output pixel value: ； wherein, is a maximum weight of the position-dependent weight function, is a set of camera perspectives covering the output pixel, is a pixel value after brightness and color correction.

8. The method of claim 1, wherein, Further comprising an automatic seam path optimization step: Constructing a seam cost function in the overlap region of the at least two aerial subgraphs; Solving the optimal seam path by using the shortest path algorithm or graph cut algorithm with the goal of minimizing the total seam path cost value; Constructing a transition band of preset width along both sides of the optimal seam path; Calculating the normalized weight of the pixels in the transition band according to their distance to the seam path, and using the γ-weighted linear blending strategy based on the normalized weight to fuse.

9. The method of claim 8, wherein, The seam cost function is expressed as: ； wherein: is the corrected pixel value at position x for the selected reference view angle; is the corrected pixel value at position x for another candidate view angle; is the gradient vector at position x for the reference view angle image, is the gradient magnitude thereof; is the structure saliency mask, taking larger values in the regions of the car body contour, lane line, parking line, and smaller values in the background regions; and a, b, g are non-negative weight coefficients for balancing the influences of the gradient term, the color difference term, and the structure saliency term.

10. An automatic driving oriented multi-fish-eye camera bird's eye view construction system for implementing the automatic driving oriented multi-fish-eye camera bird's eye view construction method of any one of claims 1-9, characterized in that, The system comprises: A reference plane construction module for establishing an aerial reference plane with unified physical scale constraints; A fisheye de-distortion module for offline calibration of each fisheye camera based on a polynomial fisheye imaging model, constructing a pixel back-projection lookup table, and performing lookup table-based de-distortion processing on the collected fisheye images; A checkerboard extraction module for extracting multi-region checkerboard corner points in the de-distorted perspective-corrected graph and the aerial reference graph, and establishing a set of cross-view corresponding points; A homography solving and optimization module for solving the initial homography matrix of each fisheye camera to the aerial reference plane based on the set of corresponding points, constructing a joint objective function containing checkerboard re-projection error, overlap region alignment error, and physical scale constraints, and performing global joint optimization on all homography matrices; An aerial mapping module for generating each aerial subgraph using the reverse mapping strategy based on the optimized homography matrix; A fusion module for identifying the seam region based on the difference value of the camera weights, using the maximum weight exclusive rendering strategy in the non-seam region, and using the γ-weighted pixel smoothing fusion in the seam region to generate the whole vehicle aerial graph.

Images