A method for detecting low obstacles using an automatic parking surround view camera

Abstract

The present invention provides a low obstacle detection method for an automatic parking surround view camera. The method collects images around a vehicle through an on-board surround view camera, annotates low obstacles in the images, and constructs a convolutional neural network training model to train the target annotated data respectively to obtain a low obstacle target pixel point set. The low obstacle target pixel point set is discretized to select key points, calculate feature descriptors of the key points, match key points of upper and lower frames using the feature descriptors, and select successfully matched key points. Finally, the position difference Q of successfully matched points in the upper and lower images is calculated in combination with vehicle motion information, and the height of the low obstacle is calculated according to the distance difference, thereby effectively detecting the type, position and height information of the low obstacle, and providing effective information for automatic parking.

Description

Technical Field

[0001] The present invention relates to the technical field of millimeter wave radars, and in particular to a method for detecting low obstacles of an automatic parking surround view camera. Background Art

[0002] Automatic parking uses the detection information of the vehicle's surround view perception and ultrasonic probe as the basis for parking control. Obstacle detection and obstacle ranging need to rely on ultrasonic waves. Large objects such as vehicles can generally be effectively identified; ground locks, wheel chocks, fences, ice cream cones and other low obstacles cannot be identified by ultrasonic waves due to insufficient ultrasonic echo information, which will seriously reduce the safety of the parking process. In addition, there is a difference between camera imaging and ultrasonic ranging, that is, ultrasonic waves are not affected by external conditions such as light, but camera imaging is significantly affected by light. In low-light conditions such as at night, it will cause imaging abnormalities, which will further cause inaccurate obstacle detection information. Summary of the invention

[0003] In view of the above problems, the purpose of the present invention is to provide a low obstacle detection method for an automatic parking surround view camera, which detects low obstacles by using deep learning target detection technology, classifies low obstacles by deep learning, and uses a monocular camera to measure distance to effectively calculate the position and height of low obstacles, thereby effectively providing accurate information for automatic parking.

[0004] Specifically, the present invention proposes a method for detecting low obstacles using an automatic parking surround view camera, comprising the following steps:

[0005] S1: The vehicle surround camera is used to collect images around the vehicle and mark low obstacles in the images.

[0006] S2: Construct a convolutional neural network model to perform semantic segmentation training and 2D target training on the target labeled data;

[0007] S3: According to the 2D target information obtained through training, a set of pixels within the 2D frame whose semantic segmentation classification probability is the same as the maximum probability of the semantic segmentation result is selected, which is the pixel set of the same low obstacle; discretize the identified low obstacle target pixel set, select key points, and calculate the feature descriptors of the key points;

[0008] S4: Use feature descriptors to match the key points of the upper and lower frames, and filter out the key points that are successfully matched;

[0009] S5: When the vehicle is moving, the position difference of the successfully matched point pairs Q in the upper and lower images is calculated based on the vehicle movement information, and the height of the low obstacle is calculated based on the position difference.

[0010] Wherein, the S1 further includes:

[0011] The semantic segmentation and labeling method is used to color-label the background, ground locks, wheel chocks, curbs, ice cream cones, parking poles and other low obstacles according to the contours of the target.

[0012] And the 2D target labeling method is used to label the target types of ground locks, wheel chocks, curbs, ice cream cones, parking poles and other low obstacles in turn according to the 2D frame.

[0013] Furthermore, the semantic segmentation training includes: obtaining a semantic segmentation annotation training set, the semantic segmentation annotation training set includes sample images and color annotation labels, generating a mask image corresponding to the sample image according to the color annotation labels according to classification information, and the sample image and the mask image remain the same size; inputting the sample image and mask image of the semantic segmentation annotation training set into a pre-built semantic segmentation model to obtain the classification probability corresponding to each pixel of the sample image; updating the parameters of the semantic segmentation model based on the classification probability and the mask image to obtain a trained semantic segmentation model.

[0014] The 2D target training includes: obtaining a 2D target sample training set, using a target detection network to process the image, and obtaining a 2D target detection result of the sample image; wherein the target detection network includes a backbone network, a feature convolution layer, a maximum pooling network, a fully connected layer, and a regional convolutional neural network RCNN; after performing a spatial search in the image according to the target detection network, performing a residual calculation with the annotated 2D box, and then iteratively updating to obtain the optimal training weight result.

[0015] The selecting of key points also includes: sorting all classified pixel coordinates from small to large according to Y values, and the maximum and minimum values ​​of X coordinates of each row after sorting are edge points, and all the selected edge points are used as key points.

[0016] The method of calculating the feature descriptor of the key point also includes: using the local feature descriptor BEBLID to calculate the local feature of each key point, and obtaining the local texture information of each key point, which is represented as a binary digitized vector V in different directions on the image, V=[V0, V1, V2, ..., Vn], where n is the dimension of the binary vector v.

[0017] The S4 also includes: establishing a corresponding relationship based on the descriptors of the two frames of images, calculating the different Hamming distances between the digitized vectors of each pair of descriptors, the Hamming distance calculation is to count the number of identical and different numbers in the two groups of binary digitized vectors, and calculating the Hamming weight w according to the identical number m: w=m / n. When w>0.8, it is considered that the two groups of points match, and the matching ones are saved as point pairs Q.

[0018] The S5 further includes:

[0019] The interval between the upper and lower images is dt. The lateral and longitudinal displacements and the longitudinal displacements dx and dy are calculated based on the wheel speed information of the vehicle. The formula is: dx = vx*dt, dy = vy*dt, where the lateral speed of the vehicle is vx and the longitudinal speed is vy; the displacements of the two groups of points in the image are dx0 and dy0, and the position difference equation is:

[0020]

[0021] Furthermore, during the movement of the vehicle, if the successfully matched point pair itself is a point on the ground, there is no position difference; if there is a position difference, it means that the matched point pair Q is not a point on the ground, and the relationship between the height of the point and the position difference is a linear relationship, and the formula is: H = a*dist, where a is the weight coefficient.

[0022] Furthermore, the coordinates of the point pair Q in the current panoramic view are (X0, Y0). Combined with the height H of the point, the true position (X, Y) of the point is calculated according to the principle of similar triangles; and the height information of all matching point pairs is sorted from small to large, and the largest point and the smallest point are selected, which are the lowest point and the highest point of the low obstacle.

[0023] In summary, the present invention provides a low obstacle detection method for an automatic parking surround view camera, which collects images around the vehicle through a vehicle-mounted surround view camera, annotates the low obstacles in the image, and constructs a convolutional neural network model to train the target annotated data respectively to obtain a low obstacle target pixel point set, discretizes the low obstacle target pixel point set, selects key points, calculates feature descriptors of the key points, and uses the feature descriptors to match the key points of the upper and lower frames, and selects the key points that are successfully matched, and finally, in combination with the vehicle motion information, calculates the position difference Q of the successfully matched point pair in the upper and lower images, and calculates the height of the low obstacle according to the position difference, thereby effectively detecting the type, position and height information of the low obstacle, and providing effective information for automatic parking. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a schematic diagram of the low obstacle detection method of the automatic parking surround view camera of the present invention.

[0025] Figure 2 1 is a test effect diagram using the method described in Figure 1. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention are clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] like Figure 1 As shown, the present invention proposes a method for detecting low obstacles by an automatic parking surround view camera, comprising the following steps:

[0028] S1: The vehicle surround camera is used to collect images around the vehicle and mark low obstacles in the images.

[0029] Specifically, the semantic segmentation and labeling method is used to color-label the background, ground locks, wheel chocks, curbs, ice cream cones, parking poles and other low obstacles in turn according to the contours of the target.

[0030] And the 2D target labeling method is used to label the target types of ground locks, wheel chocks, curbs, ice cream cones, parking poles and other low obstacles in turn according to the 2D frame.

[0031] S2: Construct a convolutional neural network model to perform semantic segmentation training and 2D target training on the target labeled data;

[0032] Specifically, the semantic segmentation training process is as follows: obtain the semantic segmentation annotation training set, which includes sample images and color annotation labels, and generate a mask map corresponding to the sample image according to the classification information of the color annotation label. The sample image and the mask map keep the same size. Input the sample image and mask map of the training set into the pre-built semantic segmentation model to obtain the classification probability corresponding to each pixel of the sample image; update the parameters of the semantic segmentation model based on the classification probability and the mask map to obtain the trained semantic segmentation model.

[0033] The 2D target training process is as follows: obtain a 2D target sample training set, use the target detection network to process the image, and obtain the 2D target detection result of the sample image. Among them, the target detection network includes a backbone network, a feature convolution layer, a maximum pooling network, a fully connected layer, and a regional convolutional neural network RCNN. The target detection network training process is to perform a spatial search in the image according to the target detection network, and then perform an iterative update after making a residual with the annotated 2D box to obtain the optimal training weight result.

[0034] S3: According to the 2D target information obtained through training, a set of pixels within the 2D frame whose semantic segmentation classification probability is the same as the maximum probability of the semantic segmentation result is selected, which is the pixel set of the same low obstacle; discretize the identified low obstacle target pixel set, select key points, and calculate the feature descriptors of the key points;

[0035] S4: Use feature descriptors to match the key points of the upper and lower frames, and filter out the key points that are successfully matched;

[0036] Specifically, the local feature descriptor BEBLID (Boosted Efficient Binary Local Image Descriptor) is used to calculate the local features of each key point to obtain the local texture information of each key point, which is represented as a binary digitized vector V in different directions on the image: V = [V0, V1, V2, ..., Vn], where n is the dimension of the binary vector v.

[0037] The correspondence is established based on the descriptors of the two frames. Using the brute force algorithm, each descriptor in the first frame is compared with all descriptors in the second frame, and the Hamming distance is used for matching, that is, the different Hamming distances between the digitized vectors of each pair of descriptors are calculated.

[0038] The calculation process of the Hamming distance is to count the number of identical and different vectors in two sets of binary digitized vectors, and calculate the Hamming weight w based on the identical number m.

[0039] w=m / n;

[0040] When w>0.8, the two groups of points are considered to match, and the matching ones are saved as point pairs Q.

[0041] S5: When the vehicle is moving, the position difference of the successfully matched point pairs Q in the upper and lower images is calculated based on the vehicle movement information, and the height of the low obstacle is calculated based on the position difference.

[0042] Specifically, the interval time between the upper and lower images is dt, and the lateral and longitudinal displacements and the longitudinal displacements dx and dy are calculated according to the wheel speed information of the vehicle in motion. The formula is: dx = vx*dt, dy = vy*dt, where the lateral speed of the vehicle is vx, and the longitudinal speed is vy; the displacements of the two groups of points in the image are dx0 and dy0, and the position difference equation is:

[0043]

[0044] During the movement of the vehicle, if the successfully matched point pair is itself a point on the ground, there is no position difference; if there is a position difference, it means that the matched point pair Q is not a point on the ground, and the relationship between the height of the point and the position difference is a linear relationship, and the formula is: H = a*dist, where a is the weight coefficient.

[0045] The coordinates of the point pair Q in the current panoramic view are (X0, Y0). Combined with the height H of the point, the actual position (X, Y) of the point is calculated according to the principle of similar triangles. The height information of all matching point pairs is sorted from small to large, and the largest and smallest points are selected, which are the lowest and highest points of the low obstacle.

[0046] Preferably, the above scheme is a process for calculating the height and position of a stationary target. For a moving target, it is necessary to first estimate the motion relationship of the moving target, and it is also applicable after motion compensation.

[0047] like Figure 2 As shown, it is a diagram showing the effect of using the method described in the present invention and after testing, wherein in the diagram, 0.0000 (mm) and 717.6146 (mm) are the minimum height and maximum height of the obstacle, as well as the positions of the minimum height and maximum height in the surround image.

[0048] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention should be based on the attached claims.

Claims

1. A method for detecting low obstacles using an automatic parking surround view camera, characterized in that: The following steps are involved: S1: The vehicle surround camera is used to collect images around the vehicle and mark low obstacles in the images. S2: Construct a convolutional neural network model to perform semantic segmentation training and 2D target training on the target labeled data; S3: According to the 2D target information obtained through training, a set of pixels within the 2D frame whose semantic segmentation classification probability is the same as the maximum probability of the semantic segmentation result is selected, which is the pixel set of the same low obstacle; discretize the identified low obstacle target pixel set, select key points, and calculate the feature descriptors of the key points; S4: Use feature descriptors to match the key points of the upper and lower frames, and filter out the key points that are successfully matched; S5: When the vehicle is moving, the position difference of the successfully matched point pairs Q in the upper and lower images is calculated based on the vehicle movement information, and the height of the low obstacle is calculated based on the position difference; The semantic segmentation training comprises: obtaining a semantic segmentation annotation training set, the semantic segmentation annotation training set comprising a sample image and a color annotation label, generating a mask image corresponding to the sample image according to the classification information of the color annotation label, and the sample image and the mask image are kept in the same size; inputting the sample image and the mask image of the semantic segmentation annotation training set into a pre-built semantic segmentation model to obtain the classification probability corresponding to each pixel of the sample image; updating the parameters of the semantic segmentation model based on the classification probability and the mask image to obtain a trained semantic segmentation model; The 2D target training includes: obtaining a 2D target sample training set, processing the image using a target detection network, and obtaining a 2D target detection result of the sample image; performing a spatial search in the image according to the target detection network, performing residual calculation with the annotated 2D box, and performing iterative updating to obtain an optimal training weight result; S5 also includes: The interval between the upper and lower images is dt. The lateral and longitudinal displacements and the longitudinal displacements dx and dy are calculated based on the wheel speed information of the vehicle. The formula is: dx = vx*dt, dy = vy*dt, where the lateral speed of the vehicle is vx and the longitudinal speed is vy; the displacements of the two groups of points in the image are dx0 and dy0, and the position difference equation is:

2. The method for detecting low obstacles using an automatic parking surround view camera according to claim 1, characterized in that: Said S1 further comprises: The semantic segmentation and labeling method is used to color-label the background, ground locks, wheel chocks, curbs, ice cream cones, parking poles and other low obstacles according to the contours of the target. And the 2D target labeling method is used to label the target types of ground locks, wheel chocks, curbs, ice cream cones, parking poles and other low obstacles in turn according to the 2D frame.

3. The method for detecting low obstacles using an automatic parking surround view camera according to claim 1, characterized in that: The target detection network includes a backbone network, a feature convolution layer, a maximum pooling network, a fully connected layer and a regional convolutional neural network RCNN.

4. The method for detecting low obstacles using an automatic parking surround view camera according to claim 1, characterized in that: The selecting of key points also includes: sorting all classified pixel coordinates from small to large according to Y values, and the maximum and minimum values ​​of X coordinates of each row after sorting are edge points, and all the selected edge points are used as key points.

5. The method for detecting low obstacles by an automatic parking surround view camera according to claim 1, characterized in that: The calculating of the feature descriptor of the key point also includes: The local feature descriptor BEBLID is used to calculate the local features of each key point to obtain the local texture information of each key point, which is represented as a binary digitized vector V in different directions on the image, V = [V0, V1, V2, ..., Vn], where n is the dimension of the binary vector v.

6. The method for detecting low obstacles using an automatic parking surround view camera according to claim 1, characterized in that: The S4 further comprises: A correspondence is established based on the descriptors of the two frames of images, and the different Hamming distances between the digitized vectors of each pair of descriptors are calculated. The Hamming distance is calculated by counting the number of identical and different vectors in two sets of binary digitized vectors. The Hamming weight w is calculated based on the identical number m: w = m / n. When w>0.8, the two sets of points are considered to match, and the matching ones are saved as point pairs Q.

7. The method for detecting low obstacles using an automatic parking surround view camera according to claim 1, characterized in that: Also includes: During the vehicle movement, if the successfully matched point pair is itself a point on the ground, there is no position difference; If there is a position difference, it means that the matched point pair Q is not a point on the ground, and the relationship between the height of the point and the position difference is a linear relationship, and the formula is: H = a*dist, where a is the weight coefficient.

8. The method for detecting low obstacles using an automatic parking surround view camera according to claim 7, characterized in that: Also includes: The coordinates of the point pair Q in the current panoramic view are (X0, Y0). Combined with the height H of the point, the actual position (X, Y) of the point is calculated according to the principle of similar triangles. The height information of all matching point pairs is sorted from small to large, and the largest and smallest points are selected, which are the lowest and highest points of the low obstacle.

Images