An automatic parking perception method fusing surround view image and lidar

Abstract

This invention relates to an automatic parking perception method integrating surround view images and lidar. The method includes the following steps: Step 1: Acquire original images using four fisheye cameras mounted on the vehicle body and correct the distortion of the original images; Step 2: Select four marker points with known ground coordinates and perform projection transformation on the distortion-corrected original images to obtain four top-view images projected onto the ground; Step 3: Stitch the four top-view images based on the spatial consistency of the marker points to obtain a surround view image; Step 4: Input the surround view image into a parking space detection model to detect parking spaces and obtain the pixel coordinates of the parking spaces; Step 5: Obtain the global coordinates of the parking spaces based on the pixel coordinates of the parking spaces for automatic parking path planning. Compared with existing technologies, this invention has advantages such as simplifying the parking space tracking algorithm, improving the intelligence and robustness of path planning, and ensuring safety during the automatic parking process.

Description

Technical Field

[0001] This invention relates to the field of automatic parking technology, and in particular to an automatic parking perception method that integrates surround view images and lidar. Background Technology

[0002] Existing automatic parking solutions mostly rely on surround view images, fusion of surround view images and millimeter-wave radar, or fusion of surround view images and ultrasonic sensors for perception information. Surround view images are used to detect parking spaces, and detection methods include traditional vision methods and deep learning methods. Millimeter-wave radar and ultrasonic sensors are used to determine whether there is a car in the target parking space and to obtain the distance of obstacles. Moreover, existing automatic parking solutions mostly use the vehicle's own kinematic information for localization and then use localization to track the coordinates of the parking space.

[0003] Parking solutions relying solely on surround-view images cannot detect obstacles. Ultrasonic sensors have a short detection range, and millimeter-wave radar has a limited detection angle and blind spots. These sensors are generally used to determine if a parking space is occupied, but can only stop and yield to obstacles. Furthermore, positioning methods that rely solely on vehicle movement can lead to cumulative errors. Therefore, for automatic parking functions, existing methods lack comprehensive environmental perception, limiting the intelligence of automatic parking and compromising safety during the process, posing significant potential risks. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and provide an automatic parking perception method that integrates surround view images and lidar.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] An automatic parking perception method that integrates surround view images and lidar, the method includes the following steps:

[0007] Step 1: Acquire raw images using four fisheye cameras mounted on the vehicle body, and then correct the distortion of the raw images;

[0008] Step 2: By selecting four marker points with known ground coordinates, project the original image after distortion correction onto the ground to obtain four aerial views.

[0009] Step 3: Based on the spatial consistency of the marker points, stitch the four overhead views together to obtain a panoramic image;

[0010] Step 4: Input the surround view image into the parking space detection model to detect parking spaces and obtain the pixel coordinates of the parking spaces;

[0011] Step 5: Obtain the global coordinates of the parking space based on the pixel coordinates of the parking space in order to plan the automatic parking path.

[0012] In step 1, four fisheye cameras are installed at the front, rear, left and right sides of the vehicle body, respectively. The fisheye camera on the left side of the vehicle body and the fisheye camera on the right side of the vehicle body are installed in the center of the left and right sides of the roof, respectively, to improve the clarity of the projected surround view image.

[0013] In step 1, the process of distortion correction of the original image specifically includes:

[0014] Distortion correction, or distortion removal, is performed on the original image. An OpenCV fisheye camera model is used to calibrate the intrinsic parameters of the four fisheye cameras, obtaining the intrinsic parameter matrix A and the distortion coefficient set K. The distortion correction transform of the image is calculated from the intrinsic parameter matrix A and the distortion coefficient set K, and the result of the distortion correction transform is represented in a mapping form. Based on the result of the distortion correction transform in the mapping form, each frame of the input original image is remapped to obtain the images corresponding to the four fisheye cameras.

[0015]

[0016] K = (k1, k2, k3, k4)

[0017] Where A is the intrinsic parameter matrix, f x and f y Let c be the focal length parameters in the x-direction and y-direction, respectively. x c represents the offset of the optical axis in the x-direction from the pixel origin. y K represents the offset of the optical axis and the pixel origin in the y-direction, K is the distortion coefficient set, and k1, k2, k3 and k4 are all distortion coefficients.

[0018] In step 2, the process of obtaining four overhead views projected onto the ground is as follows:

[0019] Two marker points were selected at the overlapping areas of the images from every two fisheye cameras, for a total of eight marker points: A1, A2, B1, B2, C1, C2, D1, and D2. The ground coordinates of the marker points were obtained through on-site measurement, and the pixel coordinates of the marker points were manually selected from the original images. The image from each fisheye camera was projected and transformed using the corresponding four marker points.

[0020] In step 3, the process of obtaining the panoramic image is specifically as follows:

[0021] A straight line is determined by the markers at the overlapping points of two adjacent top-view images. This straight line serves as the stitching line between the two adjacent top-view images. Four top-view images are stitched together using a masking method. The final panoramic image is divided into four parts using the stitching line as the boundary. Masks are created for each of the four images, and Gaussian blurring is applied to the edges of the masks to ensure a natural transition at the seams between the panoramic images. The colors of the four images are balanced to eliminate the influence of ambient light. After stitching, a 600*600 pixel panoramic image is obtained.

[0022] Step 4, the process of obtaining the pixel coordinates of the parking space, specifically includes the following steps:

[0023] Step 401: Use the YOLOv3-based parking space detection model to detect parking spaces, obtain the feature map of the surround view image, and divide the feature map of the surround view image into multiple grids;

[0024] Step 402: Regress the feature map of each grid to find the corner points of the parking spaces within the corresponding range;

[0025] Step 403: Based on the relative positional relationship of the parking space corner points, infer the location of the valid parking space. Through calculation and filtering, obtain the pixel coordinates of the four complete parking space corner points of the valid parking space. That is, one parking space corresponds to four parking space corner points. Two parking space corner points are detected. After calculation and filtering, infer the remaining two parking space corner points based on prior information.

[0026] Step 5, the process of obtaining the global coordinates of the parking space, specifically includes the following steps:

[0027] Step 501: Use a fast calibration method based on surround view image and lidar to perform calibration, obtain the parking space message slots_msg_lidar in lidar coordinate system, and perceive obstacle information in the surrounding environment in real time.

[0028] Step 502: Use LiDAR to locate the vehicle on the point cloud map, obtain the global coordinates of the vehicle location, and then form the vehicle pose information odometry for the vehicle location.

[0029] Step 503: Synchronize the timestamps of the vehicle pose message odometry and the parking space message slots_msg_lidar. Obtain the rotation and translation transformation matrix of the LiDAR coordinate system relative to the global map from the vehicle pose message odometry. Transform the radar coordinates of the parking space corner point in the LiDAR coordinate system into global coordinates in the global map to obtain the global coordinates of the parking space.

[0030] Step 504: For the global coordinates of the parking space obtained in different frames, calculate the distance between the two global coordinates in different frames, and determine whether the distance is less than the set threshold. If so, determine that the two belong to the same parking space, and calculate the average value of the global coordinates of the same parking space in different frames, and use it as the final detection result.

[0031] Step 505: Construct an obstacle grid map in real time using radar point cloud and obstacle information collected by LiDAR, select the target parking space for automatic parking from the detection results, obtain the global pose of the path planning endpoint, and then perform real-time path planning on the grid map.

[0032] Step 501, the calibration process using a rapid calibration method based on a surround view image and lidar, specifically includes the following steps:

[0033] Step 5011: During automatic parking, assuming the ground is a horizontal plane, ignoring the elevation coordinates of the radar point cloud on the ground, the process is simplified to coordinate calibration of two planes:

[0034] Step 5012: A 3x3 homography matrix H is used to describe the mapping relationship from the pixel coordinate system to the lidar coordinate system. The expression for the mapping relationship is:

[0035]

[0036] Where (x1, y1) are the pixel coordinates of the parking space corner point in the pixel coordinate system, (x2, y2) are the radar coordinates of the parking space corner point in the lidar coordinate system, and H is the homography matrix. 11 H 12 H 13 H 21 H 22 H 23 H 31 H 32 and H 33 All are elements of the homography matrix, and ∝ represents a proportional relationship;

[0037] Step 5013: Since the transformation is to homogeneous coordinates, by adjusting the elements H of the homography matrix... 33 =1 for normalization:

[0038]

[0039] Step 5014: Select calibration point pairs. Select calibration point pairs in the radar point cloud and the panoramic image respectively. Mark the calibration points by pasting reflective stickers on the ground. Find the marked points in RVIZ by displaying the reflection intensity of the radar point cloud and obtain the radar coordinates of the marked points in the lidar coordinate system. Obtain the pixel coordinates of the marked points in the panoramic image through image processing software.

[0040] Step 5015: Based on the collected calibration point pairs, solve the mapping relationship between the panoramic image plane and the lidar ground plane, that is, obtain the homography matrix H, and minimize the reprojection error;

[0041] Step 5016: After calibration, place a cone-shaped barrel at the position where the laser radar beam scans the ground, and project the radar point cloud onto the panoramic image for verification. The projection of the laser radar point cloud corresponds to the image.

[0042] Step 5017: Use the obtained homography matrix H to convert the pixel coordinates of the parking space corner point into radar coordinates in the lidar coordinate system, obtain the radar coordinates of the parking space, and publish it as a parking space message slots_msg_lidar.

[0043] In step 5015, the formula for minimizing the reprojection error is as follows:

[0044]

[0045] Where ε represents the reprojection error.

[0046] In step 5017, the formula for converting the pixel coordinates of the parking space corner point into radar coordinates in the LiDAR coordinate system is as follows:

[0047]

[0048] Where x2 is the abscissa of the parking space corner point in the lidar coordinate system, and y2 is the ordinate of the parking space corner point in the lidar coordinate system.

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] 1. The present invention proposes an automatic parking perception method that integrates surround view images and lidar, which establishes a bridge for the fusion of surround view images and point cloud perception information, fills the technical gap of lidar sensors in automatic parking applications, and realizes a more intelligent automatic parking function.

[0051] 2. This invention mounts a side camera on the roof of the vehicle, thereby improving the field of view for surround-view perception;

[0052] 3. This invention simplifies the calibration process and forms the basis for the fusion of surround view images and lidar perception information;

[0053] 4. Marking parking spaces on the global map for planning simplifies the parking space tracking algorithm, integrates LiDAR information, provides a more comprehensive perception of the environment, improves the intelligence and robustness of path planning, and also ensures safety during automatic parking. Attached Figure Description

[0054] Figure 1 This is a structural framework diagram of the present invention.

[0055] Figure 2 This is a schematic diagram of the installation position of the fisheye camera of the present invention.

[0056] Figure 3 This is a schematic diagram of the surrounding view image stitching of the present invention.

[0057] Figure 4 This is a diagram showing the calibration effect of a surround-view LiDAR.

[0058] Figure 5 This is a schematic diagram of the test results. Detailed Implementation

[0059] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0060] Example

[0061] This invention discloses an automatic parking perception method that fuses surround view images and lidar, the method comprising the following steps:

[0062] Step 1: Acquire raw images using four fisheye cameras mounted on the vehicle, and then correct the distortion of the raw images;

[0063] Step 2: By selecting four marker points with known ground coordinates, project the original image after distortion correction onto the ground to obtain four aerial views.

[0064] Step 3: Based on the spatial consistency of the marker points, stitch the four overhead views together to obtain a panoramic image;

[0065] Step 4: Input the surround view image into the parking space detection model to detect parking spaces and obtain the pixel coordinates of the parking spaces;

[0066] Step 5: Obtain the global coordinates of the parking space based on the pixel coordinates of the parking space in order to plan the automatic parking path.

[0067] like Figure 1 As shown, this invention uses four fisheye cameras with a 190-degree field of view to acquire the original image. The four fisheye cameras are respectively installed at the front, rear, left, and right sides of the vehicle body, with the left and right fisheye cameras respectively installed in the center of the left and right sides of the roof, achieving a 12×12m panoramic field of view. 2 This also improves the clarity of the panoramic image after projection transformation. The fisheye camera's installation position is as follows: Figure 2 As shown.

[0068] In step 1, distortion correction, or dedistortion, is performed on the original image acquired by the fisheye camera. The four fisheye cameras are calibrated using the OpenCV fisheye camera model to obtain the intrinsic parameter matrix A and the distortion coefficient set K. The distortion correction transformation of the original panoramic image is calculated, and the result of the distortion correction transformation is represented in a mapping form. Based on the result of the distortion correction transformation in the mapping form, a remapping transformation is performed. The same remapping transformation is performed on each subsequent frame of input image to obtain four images, thereby improving the speed of image processing.

[0069]

[0070] K = (k1, k2, k3, k4)

[0071] Where A is the intrinsic parameter matrix, f x and f y Let c be the focal length parameters in the x-direction and y-direction, respectively. x c represents the offset of the optical axis in the x-direction from the pixel origin. y K represents the offset of the optical axis and the pixel origin in the y-direction, K is the distortion coefficient set, and k1, k2, k3 and k4 are all distortion coefficients.

[0072] like Figure 3 As shown, in step 2, four marker points with known ground coordinates are selected for top-down projection transformation. Each fisheye camera image requires four marker points for projection transformation. Therefore, two marker points are selected at the overlapping areas of the field of view of every two cameras, for a total of eight marker points, namely A1, A2, B1, B2, C1, C2, D1, and D2. The ground coordinates of the marker points are obtained through on-site measurement, and the pixel coordinates of the marker points are manually selected on the image.

[0073] In step 3, a straight line is determined by the marker points at the overlapping areas of two adjacent top-view images. This straight line serves as the stitching line between the two adjacent top-view images. Four top-view images are stitched together using a mask. The stitching line is used as the boundary to divide the final panoramic image into four parts. Masks are created for each of the four images, and Gaussian blur is applied to the edges of the masks to ensure a natural transition at the seams between the panoramic images. The colors of the four images are balanced to eliminate the influence of ambient light. After stitching, a 600*600 pixel panoramic image is obtained.

[0074] In step 4, a parking space detection model based on YOLOv3 is used to detect parking spaces and obtain the feature map of the surround view image. The feature map of the surround view image is divided into a grid, and the feature map of each grid is used to regress the parking space corner points within the corresponding range. Then, based on the relative positional relationship of the parking space corner points, the location of the effective parking space is inferred, and then the pixel coordinates of the four complete parking space corner points are calculated to obtain the pixel coordinates of the parking space.

[0075] The output of parking space detection is the pixel coordinates of the parking space. When performing automatic parking path planning, it is necessary to obtain the global coordinates of the parking space in the global coordinate system and perceive the obstacle information of the surrounding environment in real time. LiDAR (LiDAR SLAM) is used to locate the vehicle on the point cloud map to obtain the global coordinates of the vehicle and form the vehicle pose information odometry. Based on the radar point cloud collected by LiDAR, an obstacle grid map is constructed in real time. Combining the vehicle positioning and the global coordinates of the corner points of the parking space, real-time path planning is performed on the grid map.

[0076] Calculate the global coordinates of the parking space on the global map:

[0077] This invention first calibrates the pixel coordinate system and the LiDAR coordinate system, namely the 2D coordinate system of the panoramic image and the 3D coordinate system of the LiDAR, to obtain the radar coordinates of the parking space corner points in the 3D coordinate system of the LiDAR. However, the calibration of the 2D coordinate system of the panoramic image and the 3D coordinate system of the LiDAR is problematic because the panoramic image is not directly acquired by the camera and lacks camera intrinsic parameters, making it impossible to calibrate using the PnP (Perspective-n-Point) method. If each fisheye camera is calibrated separately with the LiDAR coordinate system and then converted to the 2D coordinate system of the panoramic image, the calibration process becomes complex and cumbersome. Therefore, this invention… A rapid calibration method based on surround view images and LiDAR is adopted for calibration. Since parking space detection only needs to focus on the ground image information in the surround view image, the calibration process only focuses on the radar point cloud scanned by the LiDAR on the ground. During automatic parking, the ground is assumed to be a horizontal plane, and the height coordinates of the ground radar point cloud are ignored, simplifying the calibration to two plane coordinates. Therefore, a 3*3 homography matrix H is used to describe the mapping relationship from pixel coordinates to radar coordinates. Let (x1, y1) be the pixel coordinates corresponding to the corner point of the parking space in the pixel coordinate system, and (x2, y2) be the radar coordinates corresponding to the corner point of the parking space in the LiDAR coordinate system, then we have

[0078]

[0079] Where (x1, y1) are the pixel coordinates of the parking space corner point in the pixel coordinate system, (x2, y2) are the radar coordinates of the parking space corner point in the lidar coordinate system, and H is the homography matrix. 11 H 12 H 13 H 21 H 22 H 23 H 31 H 32 and H 33All are elements of the homography matrix, and ∝ represents a proportional relationship;

[0080] Since the transformation is to homogeneous coordinates, H 33 =1 for normalization:

[0081]

[0082] There are a total of 8 unknowns, therefore solving for the homography matrix H requires 8 equations. At least 4 sets of calibration point pairs need to be selected, one each in the radar point cloud and the panoramic image. Calibration points are marked by attaching reflective stickers to the ground. The marked points are located in the LiDAR coordinate system by displaying the reflection intensity of the radar point cloud in the RVIZ (Radio Vision Interchange). The pixel coordinates of the marked points in the panoramic image are obtained using image processing software. The collected calibration point pairs are used to solve for the perspective transformation relationship between the panoramic image plane and the LiDAR ground plane, thus obtaining the homography matrix H and minimizing the reprojection error. The formula for minimizing the reprojection error is:

[0083]

[0084] Where ε represents the reprojection error.

[0085] like Figure 4 As shown, after calibration, a cone-shaped barrel is placed at the position where the lidar beam scans the ground, and the lidar point cloud is projected onto the panoramic image for verification. The projection of the lidar point cloud matches the image well.

[0086] Using the homography matrix H obtained during calibration, the pixel coordinates of the parking space corner points are transformed into radar coordinates in the LiDAR coordinate system, thus obtaining the radar coordinates of the parking space, which are then published as the parking space message slots_msg_lidar.

[0087]

[0088] Where x2 is the abscissa of the parking space corner point in the lidar coordinate system, and y2 is the ordinate of the parking space corner point in the lidar coordinate system.

[0089] Obtain the vehicle pose message odometry and parking space message slots_msg_lidar from the vehicle localization, synchronize the timestamps of both, obtain the rotation and translation transformation matrix of the LiDAR coordinate system relative to the global map from the vehicle pose message odometry, transform the LiDAR coordinates of the parking space corner point in the LiDAR coordinate system into global coordinates in the global map, and obtain the global coordinates of the parking space.

[0090] like Figure 5As shown, for the global coordinates of the parking space obtained in different frames, the distance between the two global coordinates in different frames is calculated, and it is determined whether the distance is less than the set threshold. If so, it is determined that the two belong to the same parking space. For the global coordinates of the same parking space obtained in different frames, the average value of the global coordinates in different frames is calculated and used as the final detection result. The detection result is selected as the target parking space for automatic parking, and the global pose of the path planning endpoint is obtained. Then, path planning is performed.

[0091] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An automatic parking perception method integrating surround view images and lidar, characterized in that, The method includes the following steps: Step 1: Acquire raw images using four fisheye cameras mounted on the vehicle body, and then correct the distortion of the raw images; Step 2: By selecting four marker points with known ground coordinates, project the original image after distortion correction onto the ground to obtain four aerial views. Step 3: Based on the spatial consistency of the marker points, stitch the four overhead views together to obtain a panoramic image; Step 4: Input the surround view image into the parking space detection model to detect parking spaces and obtain the pixel coordinates of the parking spaces; Step 5: Obtain the global coordinates of the parking space based on the pixel coordinates of the parking space in order to plan the automatic parking path; Step 5, the process of obtaining the global coordinates of the parking space, specifically includes the following steps: Step 501: Use a fast calibration method based on surround view image and lidar to perform calibration, obtain the parking space message slots_msg_lidar in lidar coordinate system, and perceive obstacle information in the surrounding environment in real time. Step 502: Use LiDAR to locate the vehicle on the point cloud map, obtain the global coordinates of the vehicle location, and then form the vehicle pose information odometry for the vehicle location. Step 503: Synchronize the timestamps of the vehicle pose message odometry and the parking space message slots_msg_lidar. Obtain the rotation and translation transformation matrix of the LiDAR coordinate system relative to the global map from the vehicle pose message odometry. Transform the radar coordinates of the parking space corner point in the LiDAR coordinate system into global coordinates in the global map to obtain the global coordinates of the parking space. Step 504: For the global coordinates of the parking space obtained in different frames, calculate the distance between the two global coordinates in different frames, and determine whether the distance is less than the set threshold. If so, determine that the two belong to the same parking space, and calculate the average value of the global coordinates of the same parking space in different frames, and use it as the final detection result. Step 505: Construct an obstacle grid map in real time using radar point cloud and obstacle information collected by lidar, select the target parking space for automatic parking from the detection results, obtain the global pose of the path planning endpoint, and then perform real-time path planning on the grid map. Step 501, the calibration process using a rapid calibration method based on a surround view image and lidar, specifically includes the following steps: Step 5011: During automatic parking, assuming the ground is a horizontal plane, ignoring the elevation coordinates of the radar point cloud on the ground, the process is simplified to coordinate calibration of two planes: Step 5012: A 3x3 homography matrix H is used to describe the mapping relationship from the pixel coordinate system to the lidar coordinate system. The expression for the mapping relationship is: Where (x1, y1) are the pixel coordinates of the parking space corner point in the pixel coordinate system, (x2, y2) are the radar coordinates of the parking space corner point in the lidar coordinate system, and H is the homography matrix. 11 H 12 H 13 H 21 H 22 H 23 H 31 H 32 and H 33 All are elements of the homography matrix, and ∝ represents a proportional relationship; Step 5013: Since the transformation is to homogeneous coordinates, by adjusting the elements H of the homography matrix... 33 =1 for normalization: Step 5014: Select calibration point pairs. Select calibration point pairs in the radar point cloud and the panoramic image respectively. Mark the calibration points by pasting reflective stickers on the ground. Find the marked points in RVIZ by displaying the reflection intensity of the radar point cloud and obtain the radar coordinates of the marked points in the lidar coordinate system. Obtain the pixel coordinates of the marked points in the panoramic image through image processing software. Step 5015: Based on the collected calibration point pairs, solve the mapping relationship between the panoramic image plane and the lidar ground plane, that is, obtain the homography matrix H, and minimize the reprojection error; Step 5016: After calibration, place a cone-shaped barrel at the position where the laser radar beam scans the ground, and project the radar point cloud onto the panoramic image for verification. The projection of the laser radar point cloud corresponds to the image. Step 5017: Use the obtained homography matrix H to convert the pixel coordinates of the corner points of the parking space into radar coordinates in the lidar coordinate system, obtain the radar coordinates of the parking space, and publish it as a parking space message slots_msg_lidar. In step 5015, the formula for minimizing the reprojection error is as follows: Where ε represents the reprojection error; In step 5017, the formula for converting the pixel coordinates of the parking space corner point into radar coordinates in the LiDAR coordinate system is as follows: Where x2 is the x-coordinate of the parking space corner point in the lidar coordinate system, and y2 is the y-coordinate of the parking space corner point in the lidar coordinate system.

2. The automatic parking perception method based on the fusion of surround view image and lidar as described in claim 1, characterized in that, In step 1, four fisheye cameras are installed at the front, rear, left and right sides of the vehicle body, respectively. The fisheye camera on the left side of the vehicle body and the fisheye camera on the right side of the vehicle body are installed in the center of the left and right sides of the roof, respectively, to improve the clarity of the projected surround view image.

3. The automatic parking perception method based on the fusion of surround view image and lidar as described in claim 1, characterized in that, In step 1, the process of distortion correction of the original image specifically includes: Distortion correction, or distortion removal, is performed on the original image. An OpenCV fisheye camera model is used to calibrate the intrinsic parameters of the four fisheye cameras, obtaining the intrinsic parameter matrix A and the distortion coefficient set K. The distortion correction transform of the image is calculated from the intrinsic parameter matrix A and the distortion coefficient set K, and the result of the distortion correction transform is represented in a mapping form. Based on the result of the distortion correction transform in the mapping form, each frame of the input original image is remapped to obtain the images corresponding to the four fisheye cameras. K = (k1, k2, k3, k4) Where A is the intrinsic parameter matrix, f x and f y Let c be the focal length parameters in the x-direction and y-direction, respectively. x c represents the offset of the optical axis in the x-direction from the pixel origin. y K represents the offset of the optical axis and the pixel origin in the y-direction, K is the distortion coefficient set, and k1, k2, k3 and k4 are all distortion coefficients.

4. The automatic parking perception method based on the fusion of surround view image and lidar as described in claim 1, characterized in that, In step 2, the process of obtaining four overhead views projected onto the ground is as follows: Two marker points were selected at the overlapping areas of the images from every two fisheye cameras, for a total of eight marker points: A1, A2, B1, B2, C1, C2, D1, and D2. The ground coordinates of the marker points were obtained through on-site measurement, and the pixel coordinates of the marker points were manually selected from the original images. The image from each fisheye camera was projected and transformed using the corresponding four marker points.

5. The automatic parking perception method based on the fusion of surround view image and lidar as described in claim 1, characterized in that, In step 3, the process of obtaining the panoramic image is specifically as follows: A straight line is determined by the markers at the overlapping points of two adjacent top-view images. This straight line serves as the stitching line between the two adjacent top-view images. Four top-view images are stitched together using a masking method. The final panoramic image is divided into four parts using the stitching line as the boundary. Masks are created for each of the four images, and Gaussian blurring is applied to the edges of the masks to ensure a natural transition at the seams between the panoramic images. The colors of the four images are balanced to eliminate the influence of ambient light. After stitching, a 600*600 pixel panoramic image is obtained.

6. The automatic parking perception method based on the fusion of surround view image and lidar as described in claim 1, characterized in that, Step 4, the process of obtaining the pixel coordinates of the parking space, specifically includes the following steps: Step 401: Use the YOLOv3-based parking space detection model to detect parking spaces, obtain the feature map of the surround view image, and divide the feature map of the surround view image into multiple grids; Step 402: Regress the feature map of each grid to find the corner points of the parking spaces within the corresponding range; Step 403: Based on the relative positional relationship of the parking space corner points, infer the location of the valid parking space. Through calculation and filtering, obtain the pixel coordinates of the four complete parking space corner points of the valid parking space. That is, one parking space corresponds to four parking space corner points. Two parking space corner points are detected. After calculation and filtering, infer the remaining two parking space corner points based on prior information.

Images