A parking space detection method at any angle based on panoramic view

Abstract

The present invention discloses a method for detecting parking spaces at any angle based on a panoramic view. The AVM panoramic view formed by stitching pictures taken by four fisheye cameras mounted around the vehicle body is input into a high-precision lightweight small target detection network to complete feature extraction of the entire image; a parking space corner point detector is input to obtain a detection frame centered on the parking space corner point, and the feature information of the detection frame is sent to the key point detector of the next level to complete three types of corner point detection; the coordinate and direction information of the parking space entrance angle is obtained, and the last two coordinates of the parking space are calculated through the parking space geometry information to restore the entire parking space. The method of the present invention has stronger robustness and faster real-time performance, and can realize parking space recognition of any type and angle, providing a basis for deploying parking space detection technology in intelligent driving.

Description

Technical Field

[0001] The present invention belongs to the technical field of assisted driving, and in particular relates to a parking space detection method based on a panoramic view capable of realizing arbitrary parking space angle detection, which can be directly applied to the automatic parking link in assisted driving. Background Art

[0002] The automatic parking system is an important application of the active anti-collision system of automobiles in low-speed complex urban environments, and has been a research hotspot in recent years. Among them, the detection of parking spaces is undoubtedly the most basic and important part. Whether a parking space can be found and whether it can be accurately located will directly determine the success of parking. The accuracy of parking space detection is the key to the reliability of the automatic parking system, and it is also the foundation of the entire automatic parking system, and it is the prerequisite for subsequent path planning and path tracking control. With the help of different types of sensors to perceive the external parking environment, and use relevant technologies and methods to extract features, the information obtained is used to determine the space where there are no obstacles for the vehicle to park reasonably.

[0003] At present, there are many methods for detecting parking spaces. According to the category of the selected sensor, they can be mainly divided into vision-based and non-vision-based detection methods. Non-vision detection methods mainly use ultrasonic sensors, short-range radars or laser scanners. In this way, they send microwave signals to the surrounding environment and determine the distance from the sensor itself to the environment based on the time of the echo signal. This microwave sensor is independent of the lighting conditions at close range. The data processing is simple, so the speed is fast and usually real-time. But this method will completely fail when there is no car in the adjacent parking space. In contrast, vision-based detection methods are more suitable for scenes with brighter light, mainly using monocular cameras, binocular cameras, RGB-Depth cameras, and panoramic systems composed of fisheye cameras. By using cameras for image acquisition, computer vision algorithms are applied to detect parking spaces based on the captured images. These methods provide real-time visual assistance and rich image information. However, they are easily affected by environmental factors such as light and weather, have poor robustness, and can often only detect parallel or vertical parking spaces, and have low detection accuracy for inclined parking spaces. Summary of the invention

[0004] In order to solve the problems of poor robustness and real-time performance and difficulty in detecting inclined parking spaces in existing parking space detection methods, the present invention proposes an arbitrary angle parking space detection method based on panoramic view, which takes the vehicle-mounted panoramic view as input, takes parking space corner point detection as the entry point, combines deep learning algorithms such as target detection and the idea of ​​end-to-end network, and obtains an end-to-end integrated lightweight parking space detection network with both real-time and universality through network training and testing improvement, providing a basis for deploying parking space detection technology in intelligent driving. The specific technical solutions of the present invention are as follows:

[0005] A method for detecting parking spaces at any angle based on a panoramic view, wherein fisheye cameras are installed around the vehicle body, and the method comprises the following steps:

[0006] S1: Acquire the ring car image;

[0007] S2: Extract key point information to obtain the coordinates of the first two corner points of the parking space, the parking space angle and orientation information;

[0008] S3: Locate the parking space.

[0009] Furthermore, the specific process of step S1 is as follows:

[0010] S1-1: Obtain visible light images through wide-angle fisheye cameras installed on the front, rear, left and right sides of the vehicle body;

[0011] S1-2: Use the fisheye camera correction model to dedistort the visible light image;

[0012] S1-3: Perform inverse perspective transformation on the image processed in step S1-2, and transform the image to the same plane of the world coordinate system, that is, to a bird's-eye view;

[0013] S1-4: Using the common overlapping part between two adjacent frames of images, the optimal stitching direction is determined by the Levenberg-Marquard nonlinear optimization algorithm, and the images are stitched along the gap to form a 360° surround image around the vehicle body.

[0014] Furthermore, the specific process of step S2 is as follows:

[0015] S2-1: Input the surrounding vehicle image obtained in step S1 into the feature network for feature extraction;

[0016] Using the open source convolutional neural network as the network skeleton, the basic network is pre-trained on the ImageNet image data. Based on this benchmark, the feature pyramid SPP and path aggregation module PANet are built to extract network feature information at multiple scales.

[0017] S2-2: Input the network feature information obtained in step S2-1 into the corner target detector, predict the detection frame of the corner point through the two-layer convolutional network, and after training and iterative convergence of the IoU loss function, enable the corner target detector to predict the position information of the detection frame centered on the corner point based on the network feature information, including the coordinates of the center of the corner point and the length and width of the detection frame. Since the surrounding image only obtains the image information of the two corner points closest to the entire parking space and the moving vehicle, the corner target detector can only detect the position information of these two points. These two corner points are recorded as the parking space entrance corner points P1 and P2;

[0018] S2-3: Calculate the distance difference between two adjacent corner points P1 and P2:

[0019] If the distance difference is less than 3m, the parking space is a vertical parking space or an inclined parking space; if the distance difference is greater than 5m, the parking space is a parallel parking space; discard the corner points whose distance difference is neither less than 3m nor greater than 5m;

[0020] S2-4: Define three types of key points;

[0021] First, define a straight line as a parking space line that crosses the entire detection frame and has two intersections with the frame; define a ray as a starting point located inside the detection frame and has only one intersection with the detection frame;

[0022] The three key points are:

[0023] Line intersection point: The intersection point of the line and the ray in the detection frame is the starting point of the ray;

[0024] Ray intersection point: detect the intersection point of two rays in the frame;

[0025] Edge points intersecting with the edge of the detection frame: the intersection point where a straight line or ray intersects with the edge of the detection frame;

[0026] S2-5: Correspond the corner point position information obtained in step S2-2 to the network feature information in step S2-1, extract the corner point feature information corresponding to the detection box centered on the corner point, input it into the key point detector constructed according to the three types of key points in step S2-4, and obtain the category information and position coordinates corresponding to the three types of key points;

[0027] S2-6: Connect the detected straight line intersection point or ray intersection point with the edge point intersecting with the edge of the detection frame, calculate their angle, discard the points that cannot be restored to the parking space, and obtain the parking space angle α and orientation information. If α is not within the range of 90°±15°, the parking space is an inclined parking space;

[0028] Among them, each parking space includes four corner points. The parking space entrance corner points P1 and P2 are visible in the surround image, and the parking space rear corner points P3 and P4 are invisible in the surround image when the vehicle has not entered the parking space. Each parking space is composed of line segments P1P2, P2P3, P3P4, and P4P1 formed by its four corner points. The parking space angle α is the angle between the line segments formed by the parking space corner points P1P2 and P2P3. The parking space orientation information is determined by the extension direction of the straight line or ray that constitutes this parking space.

[0029] Furthermore, the specific process of step S3 is as follows:

[0030] The coordinates of the two corner points P1 and P2 at the entrance of the parking space are obtained by step S2-2, the parking space angle α and the orientation information of the corner points are obtained by step S2-5, the parking space type is determined by steps S2-3 and S2-6, and the parking space depth information d is set.i It is known that according to the geometric relationship of the parking space, the coordinate information of the rear corner points P3 and P4 of the parking space can be restored:

[0031]

[0032]

[0033] Among them, (x i ,y i ) is the position coordinates of the four corner points of the parking space in the ring image, with the center point as the origin of the coordinate axis, i = 1, 2, 3, 4;

[0034] After obtaining the four corner point information of each parking space, all the position and orientation information of the parking space can be located, providing a technical basis for parking space trajectory planning and posture control.

[0035] Furthermore, the key point detector in step S2-5 is a classification network composed of two convolutional layers, the input is corner point feature information, the output is category information and position coordinates of three types of key points, the training uses parking space images and manually annotated corner point types, and the loss function uses classification regression loss.

[0036] The beneficial effects of the present invention are:

[0037] 1. The present invention provides a method for detecting parking spaces at any angle based on a panoramic view. The method takes the vehicle-mounted panoramic view as input and parking corner point detection as the entry point. It combines deep learning algorithms such as target detection and the idea of ​​an end-to-end network to obtain an end-to-end integrated lightweight parking space detection network that is both real-time and universal.

[0038] 2. Compared with other computer vision detection algorithms, such as parking space line detection (Hough, Canny, image segmentation network, etc.) or parking space corner angle detection based on template matching, the present invention has stronger robustness and faster real-time performance, and can realize parking space recognition of any type and angle, providing a basis for deploying parking space detection technology in intelligent driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0040] Figure 1 It is a schematic diagram of the AVM shooting assistance system;

[0041] Figure 2 is a flow chart of the method of the present invention;

[0042] Figure 3 It is the definition of three types of key points based on the corner point detection frame of the present invention;

[0043] Figure 4 is a schematic diagram of the parking space angle of the present invention;

[0044] Figure 5 It is a schematic diagram of parking space restoration calculation of the present invention;

[0045] Figure 6 This is a partial corner point detection result map;

[0046] Figure 7 This is a partial corner point detection result diagram. DETAILED DESCRIPTION

[0047] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0048] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0049] like Figure 2 As shown, a method for detecting parking spaces at any angle based on a panoramic view is provided, wherein fisheye cameras are installed around the vehicle body, and the method comprises the following steps:

[0050] S1: Acquire the image of the vehicle; the specific process is as follows:

[0051] S1-1: Obtain visible light images through wide-angle fisheye cameras installed on the front, rear, left and right sides of the vehicle body, such as Figure 1 As shown;

[0052] S1-2: Use the fisheye camera correction model to dedistort the visible light image;

[0053] S1-3: Perform inverse perspective transformation on the image processed in step S1-2, and transform the image to the same plane of the world coordinate system, that is, to a bird's-eye view;

[0054] S1-4: Using the common overlapping part between two adjacent frames of images, the optimal stitching direction is determined by the Levenberg-Marquard nonlinear optimization algorithm, and the images are stitched along the gap to form a 360° surround image around the vehicle body.

[0055] S2: Extract key point information to obtain the coordinates of the first two corner points of the parking space, the parking space angle and orientation information; the specific process is as follows:

[0056] S2-1: Input the surrounding vehicle image obtained in step S1 into the feature network for feature extraction;

[0057] Using the open source convolutional neural network as the network skeleton, the basic network is pre-trained on the ImageNet image data. Based on this benchmark, the feature pyramid SPP and path aggregation module PANet are built to extract network feature information at multiple scales.

[0058] S2-2: Input the network feature information obtained in step S2-1 into the corner target detector, predict the detection frame of the corner point through the two-layer convolutional network, and after training and iterative convergence of the IoU loss function, enable the corner target detector to predict the position information of the detection frame centered on the corner point based on the network feature information, including the coordinates of the center of the corner point and the length and width of the detection frame. Since the surrounding image only obtains the image information of the two corner points closest to the entire parking space and the moving vehicle, the corner target detector can only detect the position information of these two points. These two corner points are recorded as the parking space entrance corner points P1 and P2;

[0059] The corner detection network was trained on the Tongji Parking-slot Dataset 2.0 database and verified experimentally that it can detect 99.89% of parking space corners, with an accuracy of 100%, a recall rate of 99.89%, and an image processing speed of 50fps. Some corner detection results are shown in the figure below. Figure 6-Figure 7 As shown in the figure, the corner point detection results in two different car-loop images, the box is the corner point detection box, and the number after point is the corner point detection probability.

[0060] S2-3: Calculate the distance difference between two adjacent corner points P1 and P2:

[0061] If the distance difference is less than 3m, the parking space is a vertical parking space or an inclined parking space; if the distance difference is greater than 5m, the parking space is a parallel parking space; discard the corner points whose distance difference is neither less than 3m nor greater than 5m;

[0062] S2-4: Define three types of key points;

[0063] like Figure 3 As shown, first, define a straight line as a parking space line that crosses the entire detection frame and has two intersections with the frame; a ray is defined as a starting point located inside the detection frame and has only one intersection with the detection frame;

[0064] The three key points are:

[0065] Line intersection point: The intersection point of the line and the ray in the detection frame is the starting point of the ray;

[0066] Ray intersection point: detect the intersection point of two rays in the frame;

[0067] Edge points intersecting with the edge of the detection frame: the intersection point where a straight line or ray intersects with the edge of the detection frame;

[0068] S2-5: Correspond the corner point position information obtained in step S2-2 to the network feature information in step S2-1, extract the corner point feature information corresponding to the detection box centered on the corner point, input it into the key point detector constructed according to the three types of key points in step S2-4, and obtain the category information and position coordinates corresponding to the three types of key points;

[0069] S2-6: Connect the detected straight line intersection point or ray intersection point with the edge point intersecting with the edge of the detection frame, calculate their angle, discard the points that cannot be restored to the parking space, and obtain the parking space angle α and orientation information. If α is not within the range of 90°±15°, the parking space is an inclined parking space;

[0070] Each parking space includes four corner points. The corner points P1 and P2 at the entrance of the parking space are visible in the surround image. The corner points P3 and P4 at the rear of the parking space are invisible in the surround image when the vehicle has not entered the parking space. Each parking space is composed of line segments P1P2, P2P3, P3P4, and P4P1 formed by its four corner points. Figure 4 As shown, the parking space angle α is the angle between the line segments formed by the parking space corner points P1P2 and P2P3, and the parking space orientation information is determined by the extension direction of the straight line or ray forming the parking space.

[0071] S3: Locate the parking space. The specific process is as follows:

[0072] like Figure 5 As shown, the coordinates of the two corner points P1 and P2 at the parking space entrance are obtained through step S2-2, the parking space angle α and the orientation information of the corner points are obtained through step S2-5, and the parking space type is determined through steps S2-3 and S2-6. Suppose the parking space depth information d i It is known that according to the geometric relationship of the parking space, the coordinate information of the rear corner points P3 and P4 of the parking space can be restored:

[0073]

[0074]

[0075] Among them, (x i ,y i ) is the position coordinates of the four corner points of the parking space in the ring image, with the center point as the origin of the coordinate axis, i = 1, 2, 3, 4;

[0076] After obtaining the four corner point information of each parking space, all the position and orientation information of the parking space can be located, providing a technical basis for parking space trajectory planning and posture control.

[0077] In some embodiments, the key point detector in step S2-5 is a classification network composed of two convolutional layers, the input is corner point feature information, the output is category information and position coordinates of three types of key points, the training uses parking space images and manually annotated corner point types, and the loss function uses classification regression loss.

[0078] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0079] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0080] In the present invention, the terms "first", "second", "third", "fourth" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. The term "plurality" refers to two or more, unless otherwise clearly defined.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting parking spaces at any angle based on a panoramic view, wherein fisheye cameras are installed around the vehicle body, characterized in that: The following steps are involved: S1: Acquire the ring car image; S2: Extract key point information to obtain the coordinates of the first two corner points of the parking space, the parking space angle and orientation information; S3: Locate the parking space; The specific process of step S1 is as follows: S1-1: Obtain visible light images through wide-angle fisheye cameras installed on the front, rear, left and right sides of the vehicle body; S1-2: Use the fisheye camera correction model to dedistort the visible light image; S1-3: Perform inverse perspective transformation on the image processed in step S1-2, and transform the image to the same plane of the world coordinate system, that is, to a bird's-eye view; S1-4: Using the common overlapping part between two adjacent frames of images, the optimal stitching direction is determined by the Levenberg-Marquard nonlinear optimization algorithm, and the images are stitched along the gap to form a 360° surround image around the vehicle body; The specific process of step S2 is: S2-1: Input the surrounding vehicle image obtained in step S1 into the feature network for feature extraction; Using the open source convolutional neural network as the network skeleton, the basic network is pre-trained on the ImageNet image data. Based on this benchmark, the feature pyramid SPP and path aggregation module PANet are built to extract network feature information at multiple scales. S2-2: Input the network feature information obtained in step S2-1 into the corner target detector, predict the detection frame of the corner point through the two-layer convolutional network, and after training and iterative convergence of the IoU loss function, enable the corner target detector to predict the position information of the detection frame centered on the corner point based on the network feature information, including the coordinates of the center of the corner point and the length and width of the detection frame. Since the surrounding image only obtains the image information of the two corner points closest to the entire parking space and the moving vehicle, the corner target detector can only detect the position information of these two points. These two corner points are recorded as the parking space entrance corner points P1 and P2; S2-3: Calculate the distance difference between two adjacent corner points P1 and P2: If the distance difference is less than 3m, the parking space is a vertical parking space or an inclined parking space; if the distance difference is greater than 5m, the parking space is a parallel parking space; discard the corner points whose distance difference is neither less than 3m nor greater than 5m; S2-4: Define three types of key points; First, define a straight line as a parking space line that crosses the entire detection frame and has two intersections with the frame; define a ray as a starting point located inside the detection frame and has only one intersection with the detection frame; The three key points are: Line intersection point: The intersection point of the line and the ray in the detection frame is the starting point of the ray; Ray intersection point: detect the intersection point of two rays in the frame; Edge points intersecting with the edge of the detection frame: the intersection point where a straight line or ray intersects with the edge of the detection frame; S2-5: Correspond the corner point position information obtained in step S2-2 to the network feature information in step S2-1, extract the corner point feature information corresponding to the detection box centered on the corner point, input it into the key point detector constructed according to the three types of key points in step S2-4, and obtain the category information and position coordinates corresponding to the three types of key points; S2-6: Connect the detected straight line intersection point or ray intersection point with the edge point intersecting with the edge of the detection frame, calculate their angle, discard the points that cannot be restored to the parking space, and obtain the parking space angle α and orientation information. If α is not within the range of 90°±15°, the parking space is an inclined parking space; Among them, each parking space includes four corner points. The parking space entrance corner points P1 and P2 are visible in the surround image, and the parking space rear corner points P3 and P4 are invisible in the surround image when the vehicle has not entered the parking space. Each parking space is composed of line segments P1P2, P2P3, P3P4, and P4P1 formed by its four corner points. The parking space angle α is the angle between the line segments formed by the parking space corner points P1P2 and P2P3. The parking space orientation information is determined by the extension direction of the straight line or ray that constitutes this parking space.

2. The detection method according to claim 1, characterized in that: The specific process of step S3 is: The coordinates of the two corner points P1 and P2 at the entrance of the parking space are obtained by step S2-2, the parking space angle α and the orientation information of the corner points are obtained by step S2-5, the parking space type is determined by steps S2-3 and S2-6, and the parking space depth information d is set. i It is known that according to the geometric relationship of the parking space, the coordinate information of the rear corner points P3 and P4 of the parking space can be restored: Among them, (x i ,y i ) is the position coordinates of the four corner points of the parking space in the ring image, with the center point as the origin of the coordinate axis, i = 1, 2, 3, 4; After obtaining the four corner point information of each parking space, all the position and orientation information of the parking space can be located, providing a technical basis for parking space trajectory planning and posture control.

3. The detection method according to claim 1, characterized in that: The key point detector in step S2-5 is a classification network composed of two convolutional layers, with the input being corner point feature information and the output being category information and position coordinates of three types of key points. Parking space images and manually annotated corner point types are used for training, and the loss function uses classification regression loss.

Images