A parking lot vehicle positioning method and system

Abstract

The application relates to a parking lot vehicle positioning method and system based on semantic information and image edge features, wherein the method comprises the following steps: during vehicle operation, a bird's-eye view taking a vehicle body as an original point is acquired in real time; an image segmentation neural network is used to perform image segmentation on the bird's-eye view, semantic objects in the bird's-eye view are detected, and image edge features of the segmented image are extracted; a target detection neural network is used to extract parking space information of the bird's-eye view, and a text recognition network is used to detect text labels in the extracted parking space information, so that semantic information is obtained; the image edge features and the semantic information are taken as inputs, a parking lot environment map is established, and GPS sensor trajectory data is saved; vehicle poses are calculated based on the GPS sensor trajectory data, and vehicle positioning is performed according to a priori map. Compared with the prior art, the application has the advantages of accurate positioning and strong robustness.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and in particular to a parking lot vehicle localization method and system based on semantic information and image edge features. Background Technology

[0002] In recent years, the market demand for autonomous driving has been increasing, and automated valet parking is an important application in the field. In this task, the vehicle needs to navigate into the parking lot and automatically reach the target location; therefore, accurate positioning is the most important prerequisite for its application. To achieve accurate positioning, vehicles are equipped with various sensors, such as GPS, cameras, LiDAR, and IMU, and a large number of different positioning methods have emerged, such as visual SLAM, visual-inertial SLAM, and LiDAR SLAM. To reduce sensor costs, current algorithm research mainly focuses on vision-based positioning. However, traditional visual positioning has some problems when facing automated valet parking scenarios: on the one hand, indoor and underground parking lots are mostly composed of textureless walls, pillars, and floors, which leads to unstable feature detection and matching, and traditional visual methods are prone to tracking loss; on the other hand, in parking lots, most vehicles frequently move, meaning that the prior map built by traditional vision-based SLAM algorithms cannot be used for positioning most of the time. Summary of the Invention

[0003] The purpose of this invention is to provide a parking lot vehicle localization method and system based on semantic information and image edge features. The system uses semantic features as input, which are stable over a long period of time and robust to changes in viewpoint and illumination, thus achieving a high-precision parking lot environment localization technology.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] A parking lot vehicle localization method based on semantic information and image edge features includes the following steps:

[0006] S1. During vehicle operation, obtain a bird's-eye view with the vehicle body as the origin in real time;

[0007] S2. Use an image segmentation neural network to segment the bird's-eye view, detect semantic objects in the bird's-eye view, and extract the image edge features of the segmented image;

[0008] S3. Use an object detection neural network to extract parking space information from the bird's-eye view, and use a text recognition network to detect the text identifiers in the extracted parking space information to obtain semantic information.

[0009] S4. Using image edge features and semantic information as input, build a parking lot environment map and save GPS sensor trajectory data;

[0010] S5. Calculate the vehicle pose based on GPS sensor trajectory data and locate the vehicle according to the prior map.

[0011] S1 includes the following steps:

[0012] S11. Acquire images from various angles taken by the fisheye camera mounted on the vehicle, wherein the fisheye camera is mounted on the front bumper, left and right rearview mirrors and trunk of the vehicle respectively.

[0013] S12. Stitch together images taken from various angles using a fisheye camera to create a bird's-eye view with the vehicle body as the origin.

[0014] S12 includes the following steps:

[0015] S121, Fisheye camera distortion correction;

[0016] S122. Place a calibration plate in the overlapping area of ​​the fisheye camera's field of view, and use its corner points to perform perspective transformation on the photos taken by the fisheye camera, transforming the distortion-corrected image into a top view.

[0017] S123. Find corresponding points in the overlapping areas of images taken by different cameras and calculate the affine transformation relationship between the two top views;

[0018] S124. Based on the affine transformation relationship, the four transformed top views are fused using the adaptive weighting coefficient method to obtain the bird's-eye view.

[0019] The S4 is implemented based on the SLAM algorithm, and specifically includes the following steps:

[0020] S41. Construct a pose inferrer to infer the vehicle pose at the target time based on odometer data;

[0021] S42. Using a correlation matching algorithm, a search window is established around the predicted pose. A coarse pose is calculated in the search window. Based on image edge features and semantic information, the least squares method is used to determine the degree of matching between the point cloud and the map, as well as the degree of matching between semantic objects. The coarse pose is then optimized to calculate the accurate vehicle pose.

[0022] S43. Construct a global map using sub-maps and precise vehicle poses, and optimize the global map by utilizing the matching relationships between sub-maps, poses, and semantic objects.

[0023] S5 includes the following steps:

[0024] S51. Transform the GPS sensor trajectory data into trajectory data in the site coordinate system through coordinate system transformation;

[0025] S52. Use a point cloud matching algorithm to align the site coordinate system with the coordinate system in the SLAM algorithm;

[0026] S53. Compare the distance between the current vehicle pose and the distance between each trajectory node in the prior map, and determine whether the distance is less than the threshold. If it is less than the threshold, use the current pose as the initial pose.

[0027] S54. Locate the vehicle based on the initial pose and prior map.

[0028] A parking lot vehicle localization system based on semantic information and image edge features includes:

[0029] The bird's-eye view acquisition module is used to acquire a real-time bird's-eye view with the vehicle body as the origin during vehicle operation;

[0030] The image edge feature extraction module is used to segment the bird's-eye view using an image segmentation neural network, detect semantic objects in the bird's-eye view, and extract the image edge features of the segmented image.

[0031] The semantic information extraction module is used to extract parking space information from the bird's-eye view using an object detection neural network, and to detect the text identifiers in the extracted parking space information using a text recognition network to obtain semantic information.

[0032] The map building module is used to create a parking lot environment map using image edge features and semantic information as input, and to save GPS sensor trajectory data;

[0033] The vehicle positioning module is used to calculate the vehicle's pose based on GPS sensor trajectory data and to locate the vehicle according to a prior map.

[0034] The bird's-eye view acquisition module includes:

[0035] A multi-view image acquisition module is used to acquire images from various perspectives taken by a fisheye camera mounted on the vehicle. The fisheye camera is mounted on the front bumper, left and right rearview mirrors, and trunk of the vehicle.

[0036] The image stitching module is used to stitch together images taken by the fisheye camera from various perspectives into a bird's-eye view with the vehicle body as the origin.

[0037] The image stitching module performs the following steps:

[0038] Fisheye camera distortion correction;

[0039] A calibration board is placed in the overlapping area of ​​the fisheye camera's field of view. Its corner points are used to perform perspective transformation on the photos taken by the fisheye camera, transforming the distortion-corrected image into a top view.

[0040] Find corresponding points in the overlapping area of ​​images taken by different cameras and calculate the affine transformation relationship between the two top views;

[0041] Based on the affine transformation relationship, the adaptive weighting coefficient method is used to fuse the four transformed top views to obtain the bird's-eye view.

[0042] The map building module is based on the SLAM algorithm and specifically performs the following steps:

[0043] Construct a pose inferrer to infer the vehicle pose at the target time based on odometer data;

[0044] A correlation matching algorithm is used to establish a search window around the predicted pose. A coarse pose is calculated in the search window. Based on the image edge features and semantic information, the least squares method is used to determine the degree of matching between the point cloud and the map and the degree of matching between semantic objects. The coarse pose is then optimized to calculate the accurate vehicle pose.

[0045] A global map is constructed using submaps and precise vehicle poses, and the global map is optimized by utilizing the matching relationships between submaps, poses, and semantic objects.

[0046] The vehicle positioning module performs the following steps:

[0047] The GPS sensor trajectory data is transformed into trajectory data in the site coordinate system through coordinate system transformation;

[0048] Use a point cloud matching algorithm to align the site coordinate system with the coordinate system in the SLAM algorithm;

[0049] Compare the distance between the current vehicle pose and each trajectory node in the prior map, and determine whether the distance is less than a threshold. If it is less than the threshold, use the current pose as the initial pose.

[0050] The vehicle is located based on the initial pose and prior map.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. This invention utilizes semantic features in the parking lot environment to replace visual features, thus solving the problem of unstable positioning caused by the susceptibility of traditional visual features to factors such as season, weather, and lighting.

[0053] 2. This invention constructs a 2D semantic edge feature map to solve the problem of map mismatch caused by the changing parking environment as vehicles are parked.

[0054] 3. This invention uses text data to label parking spaces, and uses this label to distinguish parking spaces with the same shape, thus solving the problem of mismatch in SLAM systems under high repetition environment. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0056] Figure 2 This is a schematic diagram illustrating the installation method of the fisheye camera in an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of the arrangement method of the calibration plate in an embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram of the image calibration results in an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram of the edge feature extraction results in an embodiment of the present invention. Detailed Implementation

[0060] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0061] This embodiment provides a parking lot vehicle localization method based on semantic information and image edge features, such as... Figure 1 As shown, it includes the following steps:

[0062] S1. During vehicle operation, acquire a bird's-eye view with the vehicle body as the origin in real time.

[0063] S11. Hardware devices such as fisheye cameras mounted on the vehicle acquire images from various angles during vehicle operation.

[0064] In this embodiment, the hardware devices include: 4 fisheye cameras, 1 switch, 1 router, 1 industrial control computer, 1 odometer, 1 GPS sensor, and several network cables. Among them, as shown... Figure 2 As shown, the fisheye cameras are mounted on the vehicle's front bumper, left and right rearview mirrors, and trunk.

[0065] S12. Stitch together images taken from various angles using a fisheye camera to create a bird's-eye view with the vehicle body as the origin.

[0066] S121. Use a distortion table to correct the distortion of a fisheye camera;

[0067] S122. Place a calibration plate in the overlapping area of ​​the fisheye camera's field of view, such as... Figure 3 As shown, perspective transformation is performed on photos taken with a fisheye camera using its corner points. Specifically, according to... Figure 3 Calculate the perspective transformation matrix for points 1, 2, 3, and 4 in the image to transform the distorted image into a top view;

[0068] S123. Find corresponding points in the overlapping areas of images captured by different cameras. In this embodiment, the method is... Figure 3 Using points 5, 6, 7, and 8 as reference points, calculate the affine transformation relationship between the two top views;

[0069] S124. Based on the affine transformation relationship, the four transformed top views are fused using the adaptive weighting coefficient method to obtain the bird's-eye view.

[0070] S2. Use an image segmentation neural network to segment the bird's-eye view, detect semantic objects in the bird's-eye view, and extract the image edge features of the segmented image.

[0071] First, the images in the dataset are labeled using calibration software, such as... Figure 4 As shown; secondly, a neural network is trained using labeled data, and the trained network model is used to segment the stitched bird's-eye view to obtain image edge features, such as... Figure 5 As shown.

[0072] S3. Use an object detection neural network to extract parking space information from the bird's-eye view, and use a text recognition network to detect the text identifiers in the extracted parking space information to obtain semantic information.

[0073] S4. Using image edge features and semantic information as input, build a 2D map of the parking lot environment and save GPS sensor trajectory data.

[0074] S4 is implemented based on the SLAM algorithm, specifically including the following steps:

[0075] S41. Construct a pose inferrer to infer the vehicle pose at the target time based on odometer data;

[0076] S42. Using a correlation matching algorithm, a search window is established around the predicted pose. A coarse pose is calculated in the search window. Based on image edge features and semantic information, the least squares method is used to determine the degree of matching between the point cloud and the map, as well as the degree of matching between semantic objects. The coarse pose is then optimized to calculate the accurate vehicle pose.

[0077] S43. Construct a global map using sub-maps and precise vehicle poses, and optimize the global map by utilizing the matching relationships between sub-maps, poses, and semantic objects.

[0078] In the target environment, the vehicle is driven through the target environment and the above steps are repeated continuously to build a map of the target environment.

[0079] S5. Calculate the vehicle pose based on GPS sensor trajectory data and locate the vehicle according to the prior map.

[0080] S51. Transform the GPS sensor trajectory data into trajectory data in the site coordinate system through coordinate system transformation;

[0081] S52. Use a point cloud matching algorithm to align the site coordinate system with the coordinate system in the SLAM algorithm;

[0082] S53. Compare the distance between the current vehicle pose and each trajectory node in the prior map, and determine whether the distance is less than the threshold. If it is less than the threshold, it is considered that the current vehicle is already in the map, and the current pose is used as the initial pose.

[0083] S54. Locate the vehicle based on the initial pose and prior map.

[0084] This embodiment also provides a parking lot vehicle positioning system based on semantic information and image edge features to implement the above method, including:

[0085] The bird's-eye view acquisition module is used to acquire a real-time bird's-eye view with the vehicle body as the origin during vehicle operation;

[0086] The image edge feature extraction module is used to segment the bird's-eye view using an image segmentation neural network, detect semantic objects in the bird's-eye view, and extract the image edge features of the segmented image.

[0087] The semantic information extraction module is used to extract parking space information from the bird's-eye view using an object detection neural network, and to detect the text identifiers in the extracted parking space information using a text recognition network to obtain semantic information.

[0088] The map building module is used to create a parking lot environment map using image edge features and semantic information as input, and to save GPS sensor trajectory data;

[0089] The vehicle positioning module is used to calculate the vehicle's pose based on GPS sensor trajectory data and to locate the vehicle according to a prior map.

[0090] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A parking lot vehicle localization method based on semantic information and image edge features, characterized in that, Includes the following steps: S1. During vehicle operation, a bird's-eye view with the vehicle body as the origin is obtained in real time; S2. Use an image segmentation neural network to segment the bird's-eye view, detect semantic objects in the bird's-eye view, and extract the image edge features of the segmented image; S3. Use an object detection neural network to extract parking space information from the bird's-eye view, and use a text recognition network to detect the text identifiers in the extracted parking space information to obtain semantic information. S4. Using image edge features and semantic information as input, a parking lot environment map is built and GPS sensor trajectory data is saved. S4 is implemented based on the SLAM algorithm. S5. Calculate the vehicle pose based on GPS sensor trajectory data and locate the vehicle according to the prior map. The S5 Includes the following steps: S51. Transform the GPS sensor trajectory data into trajectory data in the site coordinate system through coordinate system transformation; S52. Use a point cloud matching algorithm to align the site coordinate system with the coordinate system in the SLAM algorithm; S53. Compare the distance between the current vehicle pose and the distance between each trajectory node in the prior map, and determine whether the distance is less than the threshold. If it is less than the threshold, use the current pose as the initial pose. S54. Locate the vehicle based on the initial pose and prior map.

2. The parking lot vehicle localization method based on semantic information and image edge features according to claim 1, characterized in that, S1 includes the following steps: S11. Acquire images from various angles taken by the fisheye camera mounted on the vehicle, wherein the fisheye camera is mounted on the front bumper, left and right rearview mirrors and trunk of the vehicle respectively. S12. Stitch together images taken from various angles using a fisheye camera to create a bird's-eye view with the vehicle body as the origin.

3. The parking lot vehicle localization method based on semantic information and image edge features according to claim 2, characterized in that, S12 includes the following steps: S121, Fisheye camera distortion correction; S122. Place a calibration plate in the overlapping area of ​​the fisheye camera's field of view, and use its corner points to perform perspective transformation on the photos taken by the fisheye camera, transforming the distortion-corrected image into a top view. S123. Find corresponding points in the overlapping areas of images taken by different cameras and calculate the affine transformation relationship between the two top views; S124. Based on the affine transformation relationship, the four transformed top views are fused using the adaptive weighting coefficient method to obtain the bird's-eye view.

4. The parking lot vehicle localization method based on semantic information and image edge features according to claim 1, characterized in that, S4 specifically includes the following steps: S41. Construct a pose inferrer to infer the vehicle pose at the target time based on odometer data; S42. Using a correlation matching algorithm, a search window is established around the predicted pose. A coarse pose is calculated in the search window. Based on image edge features and semantic information, the least squares method is used to determine the degree of matching between the point cloud and the map, as well as the degree of matching between semantic objects. The coarse pose is then optimized to calculate the accurate vehicle pose. S43. Construct a global map using sub-maps and precise vehicle poses, and optimize the global map by utilizing the matching relationships between sub-maps, poses, and semantic objects.

5. A parking lot vehicle positioning system based on semantic information and image edge features, characterized in that, For implementing the parking lot vehicle positioning method as described in any one of claims 1-4, the system comprises: The bird's-eye view acquisition module is used to acquire a real-time bird's-eye view with the vehicle body as the origin during vehicle operation; The image edge feature extraction module is used to segment the bird's-eye view using an image segmentation neural network, detect semantic objects in the bird's-eye view, and extract the image edge features of the segmented image. The semantic information extraction module is used to extract parking space information from the bird's-eye view using an object detection neural network, and to detect the text identifiers in the extracted parking space information using a text recognition network to obtain semantic information. The map building module is used to create a parking lot environment map using image edge features and semantic information as input, and to save GPS sensor trajectory data; The vehicle positioning module is used to calculate the vehicle's pose based on GPS sensor trajectory data and to locate the vehicle according to a prior map.

6. A parking lot vehicle positioning system based on semantic information and image edge features according to claim 5, characterized in that, The bird's-eye view acquisition module includes: A multi-view image acquisition module is used to acquire images from various perspectives taken by a fisheye camera mounted on the vehicle. The fisheye camera is mounted on the front bumper, left and right rearview mirrors, and trunk of the vehicle. The image stitching module is used to stitch together images taken by the fisheye camera from various perspectives into a bird's-eye view with the vehicle body as the origin.

7. A parking lot vehicle positioning system based on semantic information and image edge features according to claim 6, characterized in that, The image stitching module performs the following steps: Fisheye camera distortion correction; A calibration board is placed in the overlapping area of ​​the fisheye camera's field of view. Its corner points are used to perform perspective transformation on the photos taken by the fisheye camera, transforming the distortion-corrected image into a top view. Find corresponding points in the overlapping area of ​​images taken by different cameras and calculate the affine transformation relationship between the two top views; Based on the affine transformation relationship, the adaptive weighting coefficient method is used to fuse the four transformed top views to obtain the bird's-eye view.

8. A parking lot vehicle positioning system based on semantic information and image edge features according to claim 5, characterized in that, The map building module is based on the SLAM algorithm and specifically performs the following steps: Construct a pose inferrer to infer the vehicle pose at the target time based on odometer data; A correlation matching algorithm is used to establish a search window around the predicted pose. A coarse pose is calculated in the search window. Based on the image edge features and semantic information, the least squares method is used to determine the degree of matching between the point cloud and the map and the degree of matching between semantic objects. The coarse pose is then optimized to calculate the accurate vehicle pose. A global map is constructed using submaps and precise vehicle poses, and the global map is optimized by utilizing the matching relationships between submaps, poses, and semantic objects.

9. A parking lot vehicle positioning system based on semantic information and image edge features according to claim 8, characterized in that, The vehicle positioning module performs the following steps: The GPS sensor trajectory data is transformed into trajectory data in the site coordinate system through coordinate system transformation; Use a point cloud matching algorithm to align the site coordinate system with the coordinate system in the SLAM algorithm; Compare the distance between the current vehicle pose and each trajectory node in the prior map, and determine whether the distance is less than a threshold. If it is less than the threshold, use the current pose as the initial pose. The vehicle is located based on the initial pose and prior map.

Images