Automatic weighting seamless splicing method for look-around BEV of autonomous vehicle

Abstract

The invention discloses an automatic weighting seamless splicing method for an all-round view BEV of an automatic driving vehicle, and belongs to the technical field of vehicle automatic driving, and the method comprises the following steps: S1, initializing four-way all-round view fisheye data, and obtaining a correction graph and correction parameters; s2, calculating a calibration plate area on the correction image, and accurately identifying calibration plate angular points on the BEV image of the area; s3, acquiring coordinates of a fisheye correction image based on the angular point coordinates identified under BEV, calculating a perspective transformation matrix of the correction image and the BEV image, and generating a fisheye BEV image before splicing; carrying out weight calculation based on a root mean square error of a coordinate conversion result; s4, pixel splicing weights of the reference image and the to-be-spliced image are determined based on the weight result, and a seamless splicing result image is generated through the images after the weights are overlaid. According to the vehicle-mounted all-round weighted seamless splicing method, weighted calculation and seamless splicing are carried out on the splicing overlapping areas, and the problem of splicing gaps can be effectively solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle automatic driving, and in particular relates to an automatic weighted seamless splicing method for an automatic driving vehicle surround view BEV. Background Art

[0002] Surround view calibration and stitching technology is one of the key technologies in the fields of autonomous driving, robot navigation, and augmented reality. The purpose of these technologies is to capture images of the surrounding environment through multiple cameras and stitch these images into a panoramic view or bird's-eye view for more comprehensive environmental perception.

[0003] The stitching result of the bird's eye view (BEV) plays a vital role in the parking process of autonomous vehicles. It can provide real-time ground information for the vehicle, and provide input for subsequent vehicle parking planning, control and obstacle avoidance functions by sensing parking space results and obstacle information on the stitching map, thus providing a basis for the realization of autonomous driving. Therefore, the stitching technology of surround view is a key technology in the field of autonomous driving.

[0004] At present, vehicle-mounted surround view calibration and stitching are mostly performed using a priori semi-calibration methods, with internal and external parameter calculations performed after the vehicle rolls off the production line. This method has good engineering and is easy to deploy. However, due to the large distortion of the surround view camera, errors are prone to occur in the recognition of the calibration area, which affects the calibration accuracy. There are differences in the calibration accuracy of the four-way surround view fisheye cameras, making it difficult to achieve seamless stitching of BEV images. Summary of the invention

[0005] In order to solve at least one of the problems raised in the above background technology, the present invention provides an automatic weighted seamless splicing method for an autonomous driving vehicle surround view BEV.

[0006] To achieve the above object, the present invention provides the following technical solution: a method for automatic weighted seamless splicing of an autonomous driving vehicle surround view BEV, comprising the following steps:

[0007] S1. Initialize the four-way surround fisheye data, obtain the correction map and correction parameters, and calculate the body coordinates of the calibration plate corner points and the BEV image coordinate parameters;

[0008] S2, based on the BEV image coordinates and correction parameters obtained in S1, calculate the calibration plate area on the correction image, calculate the perspective transformation matrix and obtain the regional BEV image, and accurately identify the calibration plate corner points on the regional BEV image;

[0009] S3, based on the coordinates of the corner points identified under BEV, obtain the coordinates of the fisheye correction image, calculate the perspective transformation matrix of the correction image and the BEV image, and generate the fisheye BEV image before stitching; perform weight calculation based on the root mean square error of the coordinate transformation result;

[0010] S4. Determine the pixel stitching weights of the reference image and the image to be stitched based on the weight results, divide and process the BEV image into the reference image according to the error value, or calculate the stitching weight of the BEV image by the pixel distance with the reference image boundary, and superimpose the weighted images to generate a seamless stitching result image.

[0011] Preferably, the specific steps of initializing the four-way surround view fisheye data, obtaining the correction map and correction parameters, and calculating the body coordinates of the calibration plate corner points and the BEV image coordinate parameters in S1 are as follows:

[0012] S11, modify the fisheye internal reference and expand the resolution of the correction result;

[0013] S12, performing distortion correction on the fisheye data based on the modified internal parameters and distortion coefficients;

[0014] S13, calculating the positions of the corner points of the chessboard under the vehicle system, and converting them into pixel space coordinates under the image under the BEV perspective through the set pixel resolution.

[0015] Preferably, in S2, the calibration plate area on the correction image is calculated, and the perspective transformation matrix of the area is calculated and the regional BEV map is obtained. The steps of further accurately identifying the corner points of the calibration plate on the regional BEV map are as follows:

[0016] S21, inversely transform the four edge coordinates of the BEV calibration plate through the correction parameters to obtain pixel coordinates on the correction image;

[0017] S22, calculating the perspective transformation matrix by using the four corner points of the pixel coordinates of the correction image and the four corner points corresponding to the BEV;

[0018] S23, perspective transforming the calibration plate area into the BEV space through the perspective transformation matrix;

[0019] S24, identifying the center checkerboard of the calibration plate in the BEV space, and obtaining the BEV pixel coordinates of the checkerboard corner points.

[0020] Preferably, in said S3, based on the coordinates of the corner points identified under BEV, the coordinates of the fisheye correction image are obtained, the perspective transformation matrix of the correction image and the BEV image is calculated, and the steps of generating the fisheye BEV image before stitching through the transformation matrix are as follows:

[0021] S31, obtaining the coordinates of the checkerboard corner points in the corrected image through inverse transformation of the perspective transformation matrix;

[0022] S32, there will be two chessboards under each fish eye, sort the corner points, select the four outermost corner points and the corresponding corner points in the BEV space, and calculate the perspective transformation matrix of the overall corrected image;

[0023] S33, performing pixel coordinate transformation on the correction image through the perspective transformation matrix to generate a BEV image of the overall correction image.

[0024] Preferably, the steps of performing weight calculation based on the root mean square error of the coordinate transformation result in S4, performing pixel stitching weighting of the reference image and the image to be stitched based on the weight result, and superimposing the weighted images to generate a seamless stitching result image are as follows:

[0025] S41, calculating the root mean square error between the coordinates of the chessboard corner points after perspective transformation and the coordinates of the real corner points calculated based on the vehicle body system;

[0026] S42, sorting the four fisheye results, selecting the image with the smallest error as the reference image, sequentially stitching the images with larger errors, and updating the reference image;

[0027] S43: For the stitching gap, weights are calculated for pixel distances from pixels of the reference image and the image to be stitched to the center line of the stitching gap, and weighted grayscale processing is performed based on the weights.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] (1) The present application provides a method for quickly locating the recognition area of ​​a vehicle-mounted surround view camera calibration plate. By optimizing the image internal parameters to obtain a larger corrected field of view, and performing a regional BEV perspective transformation on the corrected checkerboard calibration plate area, it paves the way for further accurate recognition and can effectively solve the problems of recognition failure and poor accuracy caused by large distortion during the recognition process of the surround view calibration plate.

[0030] (2) Improve the vehicle-mounted surround view weighted seamless stitching method. The weights of the four-way surround view fisheye images are calculated based on the root mean square error of the BEV corrected coordinate transformation result. The stitching order of the reference image and the image to be stitched is determined based on the weighted result. The pixel grayscale values ​​of the overlapping areas are weighted based on the stitching gap distance to achieve seamless stitching. This can effectively solve the stitching gap problem caused by the large variation in pixel values ​​of different fisheye images when using traditional stitching methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0032] Figure 1 is a flow chart of the method of the present invention;

[0033] Figure 2 This is a schematic diagram of the placement of the chessboard calibration board;

[0034] Figure 3Schematic diagram for perspective transformation corner point selection;

[0035] Figure 4 Schematic diagram of the splicing gap. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0037] The present application aims to provide a method for quickly locating the recognition area of ​​a vehicle-mounted surround view camera calibration plate to solve the problem of poor recognition accuracy caused by large surround view distortion; and to provide a vehicle-mounted surround view weighted seamless splicing method to perform weighted calculation and seamless splicing on the overlapping splicing areas to solve the splicing gap problem.

[0038] like Figure 1 As shown, an automatic weighted seamless splicing method for an autonomous driving vehicle surround view BEV of the present invention comprises the following steps:

[0039] S1. Initialize the four-way surround fisheye data, obtain the correction map and correction parameters, and calculate the body coordinates of the calibration plate corner points and the BEV image coordinate parameters;

[0040] The checkerboard calibration board is placed as follows Figure 2 As shown, the red circle is the installation position of the surround fisheye on the vehicle body;

[0041] S2, based on the BEV image coordinates and correction parameters obtained in S1, calculate the calibration plate area on the correction image, calculate the perspective transformation matrix of the area and obtain the regional BEV map, and further accurately identify the corner points of the calibration plate on the regional BEV map;

[0042] S3, based on the coordinates of the corner points identified under BEV, the coordinates of the fisheye correction image are obtained, the perspective transformation matrix of the correction image and the BEV image is calculated, and the fisheye BEV image before stitching is generated through the transformation matrix;

[0043] Weight calculation is performed based on the root mean square error of the coordinate transformation result;

[0044] S4. Based on the weight results, pixel stitching weights of the reference image and the image to be stitched are calculated. The fisheye BEV image with smaller error is used as the reference image. The stitching weight of the BEV image with larger error is calculated by the distance between the pixel and the boundary of the reference image. The weighted images are superimposed to generate a seamless stitching result image.

[0045] Furthermore, the steps of initializing the four-way surround fisheye data in S1, obtaining the correction map and correction parameters, and calculating the body coordinates of the calibration plate corner points and the BEV image coordinate parameters are as follows:

[0046] S11, modify the fisheye internal reference and expand the resolution of the correction result;

[0047] The original resolution Size0 (image_height, image_width) of the fisheye image is multiplied by 2 to expand the resolution to display more areas, generating Size1 (image_height×2, image_width×2); image_height is the number of image rows, and image_width is the number of image columns.

[0048] Fisheye internal parameter matrix Divide the focal lengths fx and fy by 2 to expand the corrected display area, multiply the focal positions cx and cy by 2 and move them to the new resolution center, and generate Among them: fx and fy are the projection scale factors of the focal length of the camera in the horizontal and vertical directions in pixel units, which represent the zoom effect of the camera imaging and are usually measured in pixels. cx and cy are the horizontal and vertical positions of the camera optical center principal point in the pixel coordinate system.

[0049] S12. Based on the modified internal parameters and distortion coefficients, the fisheye data is subjected to distortion correction.

[0050] The fisheye distortion coefficient is dist = (k 1 ,k 2 ,k 3 ,p 1 ,p 2 ), where: k 1 represents the main radial distortion, k 2 and k 3 Indicates minor radial distortion. p 1 、p 2 It is the tangential distortion parameter, which is used to describe the distortion caused by the non-parallelism between the lens and the image plane when the camera is imaging. ;

[0051] For a pixel (xu, yu) of the original image resolution Size0, the following distortion correction formula is used to obtain the point (xc, yc) of the corrected image Size1:

[0052]

[0053] r 2 =x 2 +y 2 ;

[0054]

[0055] Where: K -1 is the inverse matrix of the fisheye intrinsic parameter matrix K, (xu,yu) is the coordinate in the original image coordinate system, and the origin of the coordinate system is at the upper left corner of the image; (x,y) is the coordinate in the camera coordinate system after the inverse transformation of the image intrinsic parameter, and the origin of the coordinate system is at the image principal point (the camera optical center is at the image pixel position); r is the distance from the coordinate (x,y) to the image principal point; (xt,yt) is the coordinate in the camera coordinate system after dedistortion; K 2 is the new internal parameter matrix modified based on the internal parameter of the K matrix; (xc, yc) is the coordinate in the image coordinate system under the correction image.

[0056] S13, calculating the positions of the corner points of the chessboard under the vehicle system, and converting them into pixel space coordinates under the image under the BEV perspective through a set pixel resolution;

[0057] The pixel resolution width is set to 1024, the height is set to 1024, and the physical size of a single pixel, pixel_x and pixel_y, is set to 5 cm. Figure 2 The installation position in the following conversion formula is used to convert the coordinates (x vel ,y vel ) to (x bev ,y bev ).

[0058]

[0059] Where: (x vel ,y vel ) is the physical coordinate in the vehicle body coordinate system, pixel_x and pixel_y are the physical sizes of image pixels, and width and height are the resolutions of the image corresponding to the x-axis and y-axis in the BEV pixel space.

[0060] In S2, the calibration plate area on the correction image is calculated, and the perspective transformation matrix of the area is calculated and the regional BEV map is obtained. The steps for further accurately identifying the corner points of the calibration plate on the regional BEV map are as follows:

[0061] S21, inversely transform the four edge coordinates of the BEV calibration plate through the correction parameters to obtain pixel coordinates on the correction image;

[0062] The BEV coordinates are inversely transformed to the correction map using the following formula:

[0063]

[0064] In the specific implementation, in order to ensure that the calibration plate can be completely framed in the inverse transformation result coordinates, Figure 4 The four corner points of BEV (x bev ,y bev ) are expanded outward by 20 pixels.

[0065] S22, calculating the perspective transformation matrix by using the four corner points of the pixel coordinates of the correction image and the four corner points corresponding to the BEV;

[0066] The perspective transformation matrix P from the rectified image space to the BEV pixel space for each image i Calculate and correct the image coordinates (x rect ,y rect ) to BEV pixel coordinates (x bev ,y bev )The process is as follows:

[0067]

[0068] Through the four sets of coordinates extracted above, the matrix P can be solved i .

[0069] S23, perspective transforming the calibration plate area to the BEV space through the perspective transformation matrix, thereby improving the recognition accuracy and reducing the recognition failure rate;

[0070] In the above correction map selection range, loop each pixel coordinate and matrix P i Multiply them to obtain the local chessboard BEV perspective transformation map. This map only undergoes preliminary perspective transformation. Due to problems such as distortion, the chessboard ground is not a plane and needs further recognition.

[0071] S24, identify the chessboard at the center of the calibration plate in the BEV space, and obtain the BEV pixel coordinates of the chessboard corners; process the local chessboard BEV perspective transformation image using the OpenCV open source library function findChessboardCorners to obtain the coordinates of all the corners of the chessboard (x bev_chess ,y bev_chess )

[0072] In S3, the coordinates of the corner points are identified based on BEV, the coordinates of the fisheye correction image are obtained, and the perspective transformation matrix of the correction image and the BEV image is calculated. The steps of generating the fisheye BEV image before stitching through the transformation matrix are as follows:

[0073] S31, obtaining the coordinates of the checkerboard corner points in the corrected image through inverse transformation of the perspective transformation matrix;

[0074] By multiplying the inverse matrix of the perspective transformation matrix, the coordinates of the checkerboard corner points on the correction image (x rect_chess ,y rect_chess ):

[0075]

[0076] in: is the perspective transformation matrix P i ,(x bev_chess ,y bev_chess ) is the coordinate of the checkerboard corner point in the BEV pixel coordinate system, (x rect_chess ,y rect_chess ) are the coordinates of the checkerboard corner points in the correction image coordinate system.

[0077] S32, there will be two chessboards under each fish eye, sort the corner points, select the four outermost corner points and the corresponding corner points in the BEV space, and calculate the perspective transformation matrix of the overall corrected image;

[0078] Select two calibration plates facing the fisheye direction and extract the four extreme points of the calibration plates away from the left and right sides of the vehicle in the BEV pixel space, that is, the points shown in the four red circles, as shown in Figure 2. Figure 3 As shown;

[0079] The perspective transformation matrix P from the rectified image space to the BEV pixel space is calculated and the rectified image coordinates (x rect ,y rect ) to BEV pixel coordinates (x bev ,y bev )The process is as follows:

[0080]

[0081] The matrix P can be solved through the four sets of coordinates extracted above.

[0082] S33, performing pixel coordinate transformation on the correction image through the perspective transformation matrix to generate a BEV image of the overall correction image.

[0083] Based on the solved perspective transformation matrix P, matrix multiplication operation is performed on all pixel coordinates on the correction image Obtain the position of the overall corrected image pixels under the BEV map and generate the BEV map.

[0084] In S4, weight calculation is performed based on the root mean square error of the coordinate transformation result, pixel stitching weights of the reference image and the image to be stitched are performed based on the weight result, and the weighted images are superimposed to generate a seamless stitching result image. The steps are as follows:

[0085] S41, calculating the root mean square error between the coordinates of the chessboard corner points after perspective transformation and the coordinates of the real corner points calculated based on the vehicle body system;

[0086]

[0087] Where: rmse is the root mean square error, (x bev_chess ,y bev_chess ) is the coordinate of the corner point of the chessboard after perspective transformation, (x' bev_chess , y' bev_chess ) are the actual corner coordinates calculated based on the body system.

[0088] S42, sorting the four fisheye results, selecting the image with the smallest error as the reference image, sequentially stitching the images with larger errors, and updating the reference image;

[0089] The calculation of the centerline of the joint gap follows the intersection line of the two images. In the specific implementation process, the 45-degree angle ray of the intersection of the two images is set to calculate the coordinates of the centerline of the joint gap, such as Figure 4 The part indicated by the bold line.

[0090] S43, at the stitching gap, weights are calculated for pixel distances from pixels of the reference image and the image to be stitched to the center line of the stitching gap, and weighted grayscale processing is performed based on the weights. The higher the weight of the reference image near the center line of the stitching gap and the reference image, the lower the weight of the stitching image.

[0091] The reference image and the stitched image are iteratively weighted by image row, and the pixel stitching weight for each row is calculated as follows:

[0092] For each row of the benchmark image, the gray value gray a The weight alpha calculation formula is as follows:

[0093] alpha=(processWidth-(j-start)) / processWidth;

[0094] Among them: alpha is the gray value of each row of the reference image a The weight of processWidth is the width of the column coordinates from the stitching gap to the reference image (iterating the current processing row coordinates from the stitching gap to the grayscale gray of the reference image). a is the pixel width of the null value), j is the column coordinate on the image, and start is the column coordinate of the location of the stitching line

[0095] For the gray value of the mosaic image b The weighted beta calculation formula is as follows:

[0096] belta=1-alpha;

[0097] The gray value gray calculation formula after splicing is as follows:

[0098] gray=alpha×gray a +belta×gray b ;

[0099] Among them: gray a is the gray value of the reference image, and its weight is alpha; gray b is the gray value of the spliced ​​image, and its weight is beta; gray is the gray value after splicing.

[0100] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for automatic weighted seamless stitching of an autonomous driving vehicle surround view BEV, characterized in that: The following steps are involved: S1. Initialize the four-way surround fisheye data, obtain the correction map and correction parameters, and calculate the body coordinates of the calibration plate corner points and the BEV image coordinate parameters; S2, based on the BEV image coordinates and correction parameters obtained in S1, calculate the calibration plate area on the correction image, calculate the perspective transformation matrix and obtain the regional BEV image, and accurately identify the calibration plate corner points on the regional BEV image; S3, based on the coordinates of the corner points identified under BEV, obtain the coordinates of the fisheye correction image, calculate the perspective transformation matrix of the correction image and the BEV image, and generate the fisheye BEV image before stitching; perform weight calculation based on the root mean square error of the coordinate transformation result; S4. Determine the pixel stitching weights of the reference image and the image to be stitched based on the weight results, divide and process the BEV image into the reference image according to the error value, or calculate the stitching weight of the BEV image by the pixel distance with the reference image boundary, and superimpose the weighted images to generate a seamless stitching result image.

2. The method for automatic weighted seamless stitching of an autonomous driving vehicle surround view BEV according to claim 1, characterized in that: The specific steps of initializing the four-way surround fisheye data, obtaining the correction map and correction parameters, and calculating the body coordinates of the calibration plate corner points and the BEV image coordinate parameters in S1 are as follows: S11, modify the fisheye internal reference and expand the resolution of the correction result; S12, performing distortion correction on the fisheye data based on the modified internal parameters and distortion coefficients; S13, calculating the positions of the corner points of the chessboard under the vehicle system, and converting them into pixel space coordinates under the image under the BEV perspective through the set pixel resolution.

3. The automatic weighted seamless splicing method for surround view BEV of an autonomous driving vehicle according to claim 1, characterized in that: In S2, the calibration plate area on the correction image is calculated, and the perspective transformation matrix of the area is calculated and the regional BEV map is obtained. The steps for further accurately identifying the corner points of the calibration plate on the regional BEV map are as follows: S21, inversely transform the four edge coordinates of the BEV calibration plate through the correction parameters to obtain pixel coordinates on the correction image; S22, calculating the perspective transformation matrix by using the four corner points of the pixel coordinates of the correction image and the four corner points corresponding to the BEV; S23, perspective transforming the calibration plate area into the BEV space through the perspective transformation matrix; S24, identifying the center checkerboard of the calibration plate in the BEV space, and obtaining the BEV pixel coordinates of the checkerboard corner points.

4. The method for automatic weighted seamless stitching of surround view BEV of an autonomous driving vehicle according to claim 1, characterized in that: In the S3, based on the coordinates of the corner points identified under BEV, the coordinates of the fisheye correction image are obtained, the perspective transformation matrix of the correction image and the BEV image is calculated, and the steps of generating the fisheye BEV image before splicing through the transformation matrix are as follows: S31, obtaining the coordinates of the checkerboard corner points in the corrected image through inverse transformation of the perspective transformation matrix; S32, there will be two chessboards under each fish eye, sort the corner points, select the four outermost corner points and the corresponding corner points in the BEV space, and calculate the perspective transformation matrix of the overall corrected image; S33, performing pixel coordinate transformation on the correction image through the perspective transformation matrix to generate a BEV image of the overall correction image.

5. The method for automatic weighted seamless stitching of an autonomous driving vehicle surround view BEV according to claim 1, characterized in that: The steps of performing weight calculation based on the root mean square error of the coordinate transformation result in S4, performing pixel stitching weighting of the reference image and the image to be stitched based on the weight result, and superimposing the weighted images to generate a seamless stitching result image are as follows: S41, calculating the root mean square error between the coordinates of the chessboard corner points after perspective transformation and the coordinates of the real corner points calculated based on the vehicle body system; S42, sorting the four-way fisheye results and updating the reference image; S43: For the stitching gap, weights are calculated for pixel distances from pixels of the reference image and the image to be stitched to the center line of the stitching gap, and weighted grayscale processing is performed based on the weights.

Images