A fisheye camera-based parking environment perception method

Abstract

The application belongs to the technical field of automatic parking, and particularly relates to a parking environment sensing method. A parking environment sensing method based on a fisheye camera, steps are as follows: S1. four fisheye images around the ego vehicle at the same time are acquired; S2. obstacle detection and drivable area detection are respectively performed on the four fisheye images, and a drivable area boundary is extracted, and the drivable area boundary points are projected into a surround view top view through a homography matrix; S3. a target feature point set of an obstacle is determined; S4. the target feature point set is projected into the surround view top view; S5. noise points in the target feature points are filtered; S6. the same target feature point set under different cameras is fused; S7. the target position and direction are estimated according to different target feature point sets; S8. the determined target coordinates and drivable area boundary coordinates are converted from the surround view top view coordinate system to the ego vehicle coordinate system, and output to a downstream task. The method realizes low-cost sensing of the environment around the ego vehicle under the parking environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automatic parking, and particularly relates to a parking environment sensing method. BACKGROUND

[0002] With the gradual maturity of parking technology, how to reduce the hardware cost and improve the performance-price ratio has become a hot issue.

[0003] The environment of the commonly used fisheye camera usually adopts camera and radar fusion sensing, and the fisheye image is deformed and detected by using a deep learning model. However, this method will limit the advantage of a large fisheye camera view range, and increase the blind area around the ego vehicle. SUMMARY

[0004] The purpose of the application is to provide a low-cost solution for sensing the environment around the ego vehicle only by relying on fisheye cameras around the vehicle body.

[0005] The technical scheme of the application is a parking environment sensing method based on a fisheye camera, comprising the following steps:

[0006] S1. Obtain four fisheye images around the ego vehicle at the same time.

[0007] S2. Detect obstacles and drivable areas in the four fisheye images respectively, extract the drivable area boundary, project the drivable area boundary points to the surround overhead view through a homography matrix, and turn to S3 for obstacle detection.

[0008] S3. Determine the target feature point set of the obstacle.

[0009] S4. Project the target feature point set to the surround overhead view.

[0010] S5. Filter the noise points in the target feature points.

[0011] S6. Fuse the same target feature point set under different cameras.

[0012] S7. Estimate the target position and direction according to the target feature point set.

[0013] S8. Convert the target coordinates determined in S7 and the drivable area boundary coordinates determined in S2 from the surround overhead view coordinate system to the ego vehicle coordinate system, and output to the downstream task.

[0014] In the above scheme, specifically, in S2, the method for obtaining the drivable area boundary point set is:

[0015] S2.1. Perform edge detection on the drivable area by using a Canny edge detection algorithm.

[0016] S2.2 convert the figure into a binary figure according to the pixel value.

[0017] S2.3 set the pixel value of the edge position to 1 and the pixel value of the remaining position to 0 to obtain a drivable area boundary point set.

[0018] In the above scheme, specifically, in S3, the obstacle target feature point set is obtained according to the 2D target detection frame and the drivable area boundary, and the steps include:

[0019] S3.1 traverse the drivable area boundary points to determine whether the boundary points are located in the detection frame of a target in the fish-eye figure.

[0020] S3.2 the drivable area boundary points located in each target detection frame are taken as the feature point set of the target.

[0021] In the above scheme, specifically, the steps of S4 include:

[0022] S4.1 de-distort the target feature point set through the distortion parameters.

[0023] S4.2 project the target feature point set into the surround overhead view through the calibrated homography matrix.

[0024] In the above scheme, specifically, the steps of S5 include:

[0025] S5.1 respectively perform DBSCAN clustering on each target feature point set.

[0026] S5.2 remove the categories where noise is located, and select the categories where the number of points is greater than 30% of the total number of target feature points.

[0027] S5.3 merge the selected categories to obtain the feature point set after noise removal of the target.

[0028] In the above scheme, specifically, the steps of S6 include:

[0029] S6.1 obtain the circumscribed rectangle of the feature points.

[0030] S6.2 calculate the minimum area IOU between each two of the circumscribed rectangles of the targets in the surround overhead view coordinate system.

[0031] S6.3 set the threshold value to 0.5, merge the targets with the minimum area IOU between each two of the targets greater than 0.5, and output the merged feature point set.

[0032] In the above scheme, specifically, the steps of S7 include:

[0033] S7.1 For the target obstacle with small footprint, calculate the distance from all feature points of the obstacle to the center point of the vehicle in the surround-view top-view coordinate system, and use the feature point with the minimum distance to represent the position of the obstacle.

[0034] S7.2 For the vehicle obstacle, determine the four corner points by the following method.

[0035] S7.2.1 Obtain the maximum and minimum values of the x and y directions of all feature points of the obstacle in the surround-view top-view coordinate system, x_min_pt, x_max_pt, y_min_pt, and y_max_pt.

[0036] S7.2.2 Combine the maximum and minimum values of the x and y directions to form vectors l1, l2, l3, l4, l5, and l6.

[0037] S7.2.3 Calculate the cosine similarity between each pair of vectors l1, l2, l3, l4, l5, and l6, and obtain the minimum cosine.

[0038] S7.2.4 If the minimum cosine is less than 0.5, go to S7.2.5, or if the minimum cosine is greater than or equal to 0.5, go to S7.2.6.

[0039] S7.2.5 Obtain the corresponding vectors lm and ln, and take the intersection point of lm and ln as the reference point P of the vehicle. If the maximum value of lm and ln is less than 1.2 times the set vehicle width, take the perpendicular direction of the longer side of lm and ln as the orientation theta of the vehicle. If the maximum value of lm and ln is greater than 1.2 times the set vehicle width, take the direction of the longer side of lm and ln as the orientation theta of the vehicle.

[0040] S7.2.6 Fit all feature points of the vehicle to obtain the points P1 and P2 with the maximum distance from the vehicle. Take the point closer to the vehicle as the reference point P. If the distance between P1 and P2 is less than 1.2 times the set vehicle width, take the perpendicular direction of the line connecting P1 and P2 as the orientation theta of the vehicle. If the distance between P1 and P2 is greater than 1.2 times the set vehicle width, take the direction of the line connecting P1 and P2 as the orientation theta of the vehicle.

[0041] S7.2.7 Calculate the other three corner points of the vehicle under the conditions of the known reference point P of the vehicle, the orientation theta of the vehicle, and the set vehicle length and width.

[0042] Beneficial effects: the method realizes low-cost perception of the surrounding environment of the vehicle in a parking environment, through 2D target detection and drivable area detection on the images taken by the fisheye cameras around the vehicle, taking the points in the drivable area boundary located in the target detection frame as the target feature point set, according to the specific features of the feature point set and the category of the obstacle, finally estimating the position and attitude of the obstacle in the vehicle coordinate system. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a flowchart of the method;

[0044] Figure 2 is a schematic diagram of the result of drivable area detection in S2 of the application;

[0045] Figure 3 is a schematic diagram of the result of obstacle target feature point detection in S3 of the application;

[0046] Figure 4 is a projection ring view in S4 of the application;

[0047] Figure 5 is a schematic diagram of the result of target denoising in S5 of the application;

[0048] Figure 6 is a flowchart of S7 of the application;

[0049] Figure 7 is a schematic diagram of the labeling of obstacle feature points in S7.2.1 of the application;

[0050] Figure 8 is a schematic diagram of the connection of obstacle feature points two by two in S7.2.2 of the application;

[0051] Figure 9 is a visualization result of part of the images in the ring view top view using the method. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0053] Referring to the drawings Figure 1 A parking environment perception method based on a fisheye camera, comprising the following steps:

[0054] S1. Acquire 4 fisheye images around the vehicle at the same time.

[0055] S2. respectively on 4 fish-eye images obstacle detection and drivable area detection, extraction of drivable area boundary, the drivable area boundary point is projected to the bird's eye view through the homography matrix, for obstacle detection into S3.

[0056] Referring to the accompanying drawings Figure 2 The method for obtaining the drivable area boundary point set is:

[0057] S2.1 edge detection is carried out on the drivable area by Canny edge detection algorithm.

[0058] S2.2 the figure is converted into a binary figure according to the pixel value.

[0059] S2.3 the pixel value of the edge position is set to 1, and the pixel value of the remaining position is set to 0, to obtain the drivable area boundary point set.

[0060] S3. determine the obstacle target feature point set.

[0061] Referring to the accompanying drawings Figure 3 According to the 2D target detection frame and the drivable area boundary, the obstacle target feature point set is obtained, and the specific steps are:

[0062] S3.1 traverses the drivable area boundary point, and judges whether the boundary point is located in the detection frame of a target in the fish-eye figure;

[0063] S3.2 the drivable area boundary point located in each target detection frame is taken as the feature point set of the target.

[0064] Referring to the accompanying drawings Figure 4 S4. the target feature point set is projected into the bird's eye view.

[0065] S4.1 the target feature point set is deformed by distortion parameters;

[0066] S4.2 the target feature point set is projected into the bird's eye view through the calibrated homography matrix.

[0067] S5. filter the noise points in the target feature point set.

[0068] Since the projection into the bird's eye view adopts the inverse perspective mapping (IPM) technology which depends on the ground flatness and ground contact, objects with high height will be significantly distorted, and the obstacle target generally has high height. According to observation, the distorted part of the feature point set of the obstacle target has obvious sparseness compared with the undistorted part of the feature point set on the ground. According to this characteristic, the density-based noise application spatial clustering (DBSCAN) method is selected to remove the noise points in the parking point set, which does not need to determine the number of categories in advance and can cluster the characteristics of nonlinear relationship. The specific implementation steps are:

[0069] S5.1 DBSCAN clustering is performed on each target feature point set respectively.

[0070] S5.2 The classes with noise are removed, and the classes with the number of points greater than 30% of the total number of target feature points are selected.

[0071] S5.3 The selected classes are merged to obtain the target denoised feature point set.

[0072] The target denoised result is shown in Figure 5 .

[0073] S6. The same target feature point sets under different cameras are fused.

[0074] Due to the large viewing angle of the fisheye camera, there is an overlapping area between the four fisheye cameras of the ego vehicle, so the same object in different camera images needs to be fused, and the fusion is performed in the surround view top view coordinate system.

[0075] The feature point set is calculated to obtain the circumscribed rectangle, and the threshold value is set to fuse the feature point sets of the same target under different cameras. The specific steps are as follows:

[0076] S6.1 Calculate the circumscribed rectangle of the feature points.

[0077] S6.2 Calculate the minimum area IOU between each pair of target circumscribed rectangles in the surround view top view coordinate system.

[0078] S6.3 Set the threshold value to 0.5, merge the targets with the minimum area IOU greater than 0.5 between each pair of targets, and output the merged feature point set.

[0079] Referring to the accompanying Figure 6 , S7. The target position and direction are estimated according to the feature point set of different targets.

[0080] S7.1 For cone barrels, columns, people and other small-area obstacles, calculate the distance from all feature points of the obstacle in the surround view top view coordinate system to the center point of the vehicle, and use the feature point corresponding to the minimum distance to represent the position of the obstacle.

[0081] S7.2 For vehicle obstacles, the four corner points are determined by the following method.

[0082] Referring to the accompanying Figure 7 , S7.2.1 Obtain the maximum and minimum values of the x and y directions of all feature points of the obstacle in the surround view top view coordinate system, x_min_pt, x_max_pt, y_min_pt, and y_max_pt.

[0083] Referring to the accompanying Figure 8S7.2.2 combine x_min_pt, x_max_pt, y_min_pt, y_max_pt pairwise to form vectors l1, l2, l3, l4, l5, l6.

[0084] S7.2.3 calculate cosine similarity cosine between l1, l2, l3, l4, l5, l6 pairwise to obtain minimum cosine.

[0085]

[0086]

[0087] i,j∈[1,6],i≠j

[0088] S7.2.4 if minimum cosine is less than 0.5, go to S7.2.5, or if minimum cosine is greater than or equal to 0.5, go to S7.2.6.

[0089] S7.2.5 obtain corresponding vectors lm, ln; take intersection of lm, ln as reference point P of the vehicle; if maximum value of lm, ln is less than 1.2 times of set vehicle width, take vertical direction of longer side of lm, ln as vehicle orientation theta, if maximum value of lm, ln is greater than 1.2 times of set vehicle width, take direction of longer side of lm, ln as vehicle orientation theta.

[0090] S7.2.6 fit all feature points of the vehicle to obtain points P1, P2 with maximum distance of vehicle point set; take point closer to vehicle from P1, P2 as reference point P; if distance of P1, P2 is less than 1.2 times of set vehicle width, take vertical direction of line connecting P1, P2 as vehicle orientation theta, if distance of P1, P2 is greater than 1.2 times of set vehicle width, take direction of line connecting P1, P2 as vehicle orientation theta.

[0091] S7.2.7 calculate other three corner points of the vehicle under the condition of known vehicle reference point P, vehicle orientation theta, and set vehicle length and width L, W.

[0092] Take a certain situation of left parking space as an example, calculation method of other three corner points P1, P2, P3 is as follows:

[0093] P1=(P x -L*||cos(theta)||,P y +L*||sin(theta)||)

[0094] P2=(P x +W*||cos(theta)|,P y+ W * || sin(theta) || )

[0095] P3 = (P1 x + W * || cos(theta) ||, P1 y + W * || | sin(theta) || )

[0096] S8. Convert the target coordinates determined in S7 and the drivable area boundary coordinates determined in S2 from the surround-view top-view coordinate system to the ego-vehicle coordinate system, and output to the downstream task.

[0097] Although the present application has been described in detail with general description and specific embodiments above, some modifications or improvements can be made on the basis of the present application, which is obvious to those skilled in the art. Therefore, these modifications or improvements made on the basis of not deviating from the spirit of the present application, all belong to the scope of the present application claimed.

Claims

1. A fisheye camera-based parking environment perception method, characterized in that, Comprise the following steps: S1. Obtain 4 fisheye images around the vehicle at the same time; S2. Respectively, the 4 fisheye images are detected and the drivable area is detected, the drivable area boundary is extracted, the drivable area boundary point is projected into the surround view top view through the homography matrix, and the obstacle detection is turned into S3; S3. Determine the target feature point set of the obstacle; S4. Project the target feature point set into the surround view top view; S5. Filter the noise points in the target feature points; S6. Fusion is carried out on the same target feature point set under different cameras; S7. According to the feature point set of different targets, the position and direction of the target are estimated; S7.1 For the target obstacle with small area, calculate the distance from all feature points of the obstacle in the surround view top view coordinate system to the vehicle center point, and use the feature point corresponding to the minimum distance to represent the position of the obstacle; S7.2 For the vehicle obstacle, four corner points are determined by the following method; S7.2.1 Obtain the maximum and minimum values of all feature points x, y of the obstacle in the surround view top view coordinate system x_min_pt, x_max_pt, y_min_pt, y_max_pt; S7.2.2 Combine the x_min_pt, x_max_pt, y_min_pt, y_max_pt two by two to form vectors l1, l2, l3, l4, l5, l6; S7.2.3 Calculate the cosine similarity between l1, l2, l3, l4, l5, l6, and obtain the minimum cosine; S7.2.4 If the minimum cosine is less than 0.5, turn to S7.2.5, or if the minimum cosine is greater than or equal to 0.5, turn to S7.2.6; S7.2.5 Obtain the corresponding vectors lm, ln, and take the intersection of lm, ln as the reference point P of the vehicle. If the maximum value of lm, ln is less than 1.2 times the set vehicle width, take the vertical direction of the longer side of lm, ln as the direction of the vehicle theta, if the maximum value of lm, ln is greater than 1.2 times the set vehicle width, take the direction of the longer side of lm, ln as the direction of the vehicle theta; S7.2.6 Fit all feature points of the vehicle to obtain the points P1, P2 with the maximum distance; set the point closer to the vehicle as the reference point P; if the distance between P1 and P2 is less than 1.2 times the set vehicle width, take the vertical direction of the line connecting P1 and P2 as the direction of the vehicle theta, if the distance between P1 and P2 is greater than 1.2 times the set vehicle width, take the direction of the line connecting P1 and P2 as the direction of the vehicle theta; S7.2.7 Calculate the other three corner points of the vehicle under the condition of knowing the vehicle reference point P, the vehicle direction theta and the set vehicle length and width; S8. Convert the target coordinates determined in S7 and the drivable area boundary coordinates determined in S2 from the surround view top view coordinate system to the vehicle coordinate system, and output to the downstream task. 2.The fisheye camera based parking environment perception method of claim 1, wherein, In S2, the method for obtaining the drivable area boundary point set is: S2.1 edge detection of the drivable area by Canny edge detection algorithm; S2.2 convert the figure into a binary image according to the pixel value; S2.3 set the pixel value of the edge position to 1 and the pixel value of the remaining position to 0 to obtain the boundary point set of the drivable area.

3. The fisheye camera-based parking environment perception method of claim 1 or 2, wherein, In S3, the obstacle target feature point set is obtained according to the 2D target detection frame and the drivable area boundary, and the steps include: S3.1 traverse the boundary points of the drivable area, and judge whether the boundary points are located in the detection frame of a target in the fisheye figure; S3.2 the boundary points of the drivable area located in each target detection frame are taken as the feature point set of the target.

4. The fisheye camera-based parking environment perception method of claim 1 or 2, wherein, The steps of S4 include: S4.1 remove the distortion of the target feature point set by the distortion parameters; S4.2 project the target feature point set into the surround view top view by the calibrated homography matrix.

5. The fisheye camera-based parking environment perception method of claim 1 or 2, wherein, The steps of S5 include: S5.1 respectively cluster each target feature point set by DBSCAN clustering; S5.2 remove the classes with noise and select the classes with more than 30% of the total number of target feature points; S5.3 merge the selected classes to obtain the feature point set after denoising of the target.

6. The fisheye camera-based parking environment perception method of claim 1 or 2, wherein, The steps of S6 include: S6.1 calculate the circumscribed rectangle of the feature points; S6.2 calculate the minimum area IOU between each two of the circumscribed rectangles of all targets in the surround view top view coordinate system; S6.3 set the threshold to 0.5, merge the targets with the minimum area IOU between each two of which greater than 0.5, and output the merged feature point set.

Images