Vision-based spatial parking space detection method and system, and automobile

Abstract

The present invention provides a vision-based spatial parking space detection method, comprising: after performing brightness pre-processing on a surround-view image, inputting the image into a trained visual parking space instance segmentation model, so as to obtain first-stage available parking space instance regions; for each first-stage available parking space instance region, performing dilation followed by a corrosion operation, then performing area screening, and removing instance regions having a pixel area less than a theoretical minimum pixel area for a parking space, while using the remaining regions as second-stage available parking space instance regions; selecting instance regions closest to preset locations as third-stage available parking space instance regions; and constructing a maximum instance boundary of a spatial parking space, and on this basis, constructing an available spatial parking space.

Description

A vision-based spatial parking space detection method, system and vehicle

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The embodiments of this application are based on the Chinese patent application with application number 202311140496.8 and application date September 4, 2023, and claim the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into the embodiments of this application as a reference. Technical Field

[0003] The present invention belongs to the field of automatic parking assistance, and in particular relates to a vision-based parking space detection method, system and vehicle. Background Art

[0004] Technologies related to automatic parking assistance are constantly developing and improving, from the initial ultrasonic radar-based spatial parking (APA) to the later mainstream visual spatial parking fusion automatic parking assistance (FAPA), and now the industry-leading on-site memory parking assistance (LAPA), and the future valet fully automatic parking assistance solution (AVP).

[0005] However, the spatial parking space detection in the FAPA / LAPA / AVP schemes still primarily relies on ultrasonic radar, as used in the original APA scheme. This method uses ultrasonic radar to detect obstacle distances during vehicle travel and then constructs a logical space for parking by identifying obstacle boundary points based on sudden changes in the ultrasonic radar distance. Due to the limitations of ultrasonic radar detection accuracy, ultrasonic radar has poor accuracy and cannot detect diagonal parking spaces. For example, the boundary error of ultrasonic radar detection is generally in the decimeter range. Ultrasonic radar can only detect parking spaces perpendicular or parallel to the vehicle body and cannot detect inclined spaces. Furthermore, due to the height restrictions of ultrasonic sensors, ultrasonic radar cannot detect low obstacles or the flatness of the ground, resulting in false positives. For example, spaces may be constructed above low flower beds and lawns, or near low obstacles, leading to illegal parking and even chassis scratches.

[0006] Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a vision-based spatial parking space detection method, system and vehicle to improve the accuracy of spatial parking space detection.

[0008] The technical solution adopted by the present invention to solve the above technical problems is: a vision-based spatial parking space detection method is applied to the electronic equipment of the vehicle,

[0009] Receive the surround view image of the vehicle and perform brightness preprocessing on the surround view image;

[0010] The surround view image after brightness preprocessing is input into the trained visual parking space instance segmentation model to obtain the first-level parking space instance area in the surround view image;

[0011] Performing a dilation and then an erosion operation on each of the first-level parking space instance regions, and then performing area screening to remove instance regions whose pixel areas are smaller than the theoretical minimum pixel area of ​​a parking space. The remaining first-level parking space instance regions are used as second-level parking space instance regions.

[0012] Selecting, from the second-level parking space instance areas, an instance area closest to a preset position as a third-level parking space instance area;

[0013] For the three-level parking space instance area, construct the maximum instance boundary of the space parking space;

[0014] According to the required parking space size, the parking space is constructed based on the maximum instance boundary.

[0015] As a second aspect, the present invention provides a vision-based spatial parking space detection system, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method when executing the program.

[0016] As a third aspect, the present invention further provides a car, comprising a surround-view camera and the vision-based spatial parking space detection system.

[0017] As a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the method when executed by a processor.

[0018] The beneficial effects of the present invention are as follows: the present invention is based on vision, using the surround-view images captured and stitched by the vehicle's built-in surround-view camera as its basis, and performs a preliminary screening of available parking space instances based on an instance segmentation algorithm. Then, a dilation-then-erosion operation is performed to eliminate small holes in the parkable space instance area. A secondary screening is then performed using area comparison. Finally, the maximum instance boundary of the spatial parking space is constructed based on image processing, thereby constructing the available parking space, thereby improving the accuracy of available parking space detection. At the same time, the system directly uses the vehicle's own surround-view images, without requiring additional hardware costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG1 is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described below with reference to specific examples and accompanying drawings.

[0021] The present invention provides a vision-based parking space detection method, as shown in Figure 1. This method is used in a car with a surround-view camera and is triggered when a signal to start the parking space search function is received. If no parking space that meets the requirements is detected, the search continues until a parking space is found or a signal to exit the parking space search function is received. The signal to start and exit the parking space search function can be set according to actual conditions. For example, it can be a touch button or a physical button on the vehicle human-machine interface or the steering wheel, or a linkage scheme such as linkage with the reverse gear. This method specifically includes the following steps:

[0022] S1. Receive the vehicle's surround view image and perform brightness preprocessing on it. This surround view image refers to a complete, stitched image, typically composed of images captured by four surround view cameras (fisheye cameras) located in the front, rear, left, and right corners of the vehicle. This image stitching is performed by the surround view system; this method only obtains the surround view image from the system. The purpose of brightness preprocessing is to correct image brightness, reduce noise, and mitigate the impact of subsequent parking space instance segmentation on the surround view image.

[0023] In this embodiment, brightness preprocessing specifically includes: calculating the average brightness of the surround-view image; classifying the surround-view image into an ultra-bright picture, a high-brightness picture, a normal-brightness picture, a dark picture, or an ultra-dark picture based on the average brightness and a preset threshold; performing brightness correction on surround-view images other than the normal-brightness picture so that the average brightness of the surround-view image is within the normal-brightness picture range; and then performing median filtering.

[0024] S2. Input the surround view image after brightness preprocessing into the trained visual parking space instance segmentation model to obtain first-level parking space instance regions in the surround view image. The first-level parking space instance regions are several annotated pixel blocks in the surround view image. The pixels in each pixel block are annotated. Pixels in the same pixel block correspond to the same parking space instance region, while pixels in different pixel blocks correspond to different parking space instance regions.

[0025] The visual parking space instance segmentation model adopts an instance segmentation algorithm and is trained using a surround-view parking space instance segmentation dataset.

[0026] Specifically, we first established a surround-view parking space instance segmentation dataset. We collected a large number of surround-view parking images and screened them. Only those images with at least one reasonably available parking space were retained as valid images in the dataset. We then annotated the parking space instances. The basic annotation criteria were that only reasonable parking space instances were annotated, and that at least one side of the parking instance region had an obstacle. The surround-view parking images referred to here were stitched surround-view images taken during the parking process.

[0027] After the dataset is prepared, an instance segmentation model such as Mask R-CNN is used for model training to obtain a visual parking space instance segmentation model.

[0028] S3. Perform a dilation followed by an erosion operation on each of the first-level parking space instance regions. Then, perform area screening to remove instance regions whose pixel areas are smaller than the theoretical minimum pixel area of ​​a parking space. The remaining first-level parking space instance regions are used as second-level parking space instance regions.

[0029] This is done because, due to image noise and instance segmentation accuracy limitations, individual pixels in the first-level parking space instance regions output by the visual parking space instance segmentation model may still fail to be segmented, resulting in small holes in the segmented parking space instance regions. Therefore, a dilation-then-erosion operation is performed on the first-level parking space instance regions. This eliminates these small holes in the parking space instance regions detected by the model without significantly changing the area and boundaries of the first-level parking space instance regions. The dilation and erosion operations described above are image processing techniques.

[0030] Subsequently, the areas of the parking space instances obtained by dilation and erosion are screened. If the pixel area (or number of pixels) of the instance areas is smaller than the theoretical minimum pixel area of ​​a parking space, these instances that do not meet the requirements are removed. The remaining first-level parking space instances are listed as candidate areas for the second-level parking space instances. The theoretical minimum pixel area of ​​a parking space can be preset based on the vehicle's situation.

[0031] S4. Selecting an instance area closest to the preset position from the second-level parking space instance areas as a third-level parking space instance area.

[0032] Since most obstacles in parking lots are 3D, they can block the available parking spaces in non-central areas of the surround view image, affecting the calculation of the available parking space boundaries. To reduce the impact of 3D obstacles on the instance segmentation of the available parking space within the surround view image, this embodiment selects the instance areas closest to the vehicle's left and right surround view cameras and close to the center of the surround view image as the third-level available parking space instance areas. This is determined by the following method:

[0033] Calculate the center coordinates of each secondary parking space instance using the center of the vehicle's rear axle as the origin, the vehicle's front direction as the y-direction, and the direction perpendicular to the right side of the vehicle as the x-direction.

[0034] The secondary parking space instance area is divided into the left and right sides of the vehicle body according to the central coordinate point, and the weighted distance L from the secondary parking space instance area on the left and right sides to the instance screening of the left and right surround view cameras is calculated respectively. l and L r :

[0035] Where x l 、y l is the x and y coordinates of the surround view camera on the left side of the vehicle body, x r ,y r is the x and y coordinates of the surround view camera on the right side of the vehicle body, x i 、y i are the x and y coordinates of each pixel in a second-level parking space instance area, n is the total number of pixels in a second-level parking space instance area, and a is a preset value;

[0036] Calculate L separately l and L r The minimum value of is used to obtain the three-level parking space instance area.

[0037] This embodiment selects the instance area closest to the vehicle's left and right surround-view cameras and near the center of the surround-view image, rather than the instance area closest to the front and rear surround-view cameras. This is because the surround-view cameras are fisheye cameras, which may cause misalignment and distortion in the diagonal front and rear during image acquisition and stitching. Therefore, it is better to select the instance area closest to the vehicle's left and right surround-view cameras. Furthermore, the closer to the center of the surround-view image, the less likely it is to be distorted.

[0038] S5. Construct the maximum instance boundary of the three-level parking space instance area. Since the target parking spaces to be constructed include vertical, horizontal, and diagonal spaces, this embodiment proposes a method for constructing a bidirectionally preferred maximum inscribed trapezoidal parking space instance boundary. This method first considers the depth of the instance space to construct the instance boundary, finding the maximum instance boundary for constructing vertical and diagonal spaces. It then considers the width of the instance space to construct the instance boundary, finding the maximum instance boundary for constructing horizontal spaces.

[0039] A bidirectional preference maximum inscribed trapezoidal parking space instance boundary construction method is adopted. First, the vertical depth of the instance space is considered to construct the instance boundary, so as to find the maximum instance boundary that can construct vertical space parking spaces and diagonal space parking spaces; then the horizontal width of the instance space is considered to construct the instance boundary, so as to find the maximum instance boundary that can construct horizontal space parking spaces. This improves the detection capability of diagonal parking spaces, and the computational complexity of this construction method is small.

[0040] The length of the bottom side of the trapezoid close to the vehicle is L1, and the length of the corresponding side of the trapezoid on the other end is L2. These two sides are parallel to each other, and the height of the trapezoid is H. (L1,L2) , the vertical priority instance boundary trapezoid construction formula is obtained as follows:

[0041] Where W H is the corrected vertical priority instance boundary trapezoidal area, α is the correction parameter;

[0042] The formula for constructing the horizontal priority instance boundary trapezoid is:

[0043] Where W L is the corrected trapezoidal area of ​​the lateral priority instance boundary, and β is the lateral correction parameter;

[0044] W found separately H and W L The maximum value is obtained to obtain the vertical priority instance trapezoidal boundary and the horizontal priority instance trapezoidal boundary.

[0045] S6. Construct a parking space based on the maximum instance boundary according to the required parking space size.

[0046] Based on the longitudinal priority trapezoidal boundary and the transverse priority trapezoidal boundary, a method for constructing a parking space is as follows:

[0047] First, a standard vertical space parking space is constructed in the trapezoidal boundary of the longitudinal priority instance. If it is satisfied, a standard vertical space parking space is constructed; the size of the standard vertical space parking space is preset.

[0048] Assume that the waist of the trapezoidal boundary closer to the rear of the vehicle is the first side of the trapezoidal boundary. When the longitudinal priority instance trapezoidal boundary does not meet the requirements for constructing a standard vertical space parking space, calculate the angle between the first side of the longitudinal priority instance trapezoidal boundary and the bottom side of the trapezoid on the side of the trapezoidal boundary close to the vehicle. Use this angle and the first side of the trapezoidal boundary as the basis to construct a standard-sized oblique parking space. If the space meets the requirements, construct a standard oblique space parking space. The size of the standard oblique space parking space is preset.

[0049] If the construction of standard vertical space parking space and standard oblique space parking space is unsuccessful, a standard horizontal space parking space is constructed in the trapezoidal boundary of the horizontal priority instance. The standard horizontal space parking space is parallel to the bottom edge of the trapezoidal boundary of the horizontal priority instance. If the space meets the requirements, a standard horizontal space parking space is constructed; the size of the standard horizontal space parking space is preset.

[0050] If the conditions for constructing a standard horizontal space parking space are still not met, there is currently no suitable parking space available.

[0051] When there is a parking space available, the system can perform automatic parking based on this parking space detection method, that is, select the parking space, start automatic parking, and end after parking is completed.

[0052] The present invention also provides a vision-based spatial parking space detection system, comprising a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the aforementioned method is implemented. The memory can be storage space integrated into a controller or an external storage medium usable on a vehicle. The processor can be an integrated or external data processing chip, such as a vehicle controller, that can be used on a vehicle. The program can be program code that can be executed on the processor.

[0053] The present invention also provides a car, comprising a surround-view camera and the vision-based spatial parking space detection system.

[0054] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the method when executed by a processor.

[0055] The present invention adopts an instance segmentation algorithm, which can more directly find candidate areas for constructing spatial parking spaces and sort instances, reducing the impact of non-parking drivable areas on the construction of spatial parking spaces. Then, the parking space instances are screened three times through image processing, and the scope is gradually narrowed. During the screening, the weighted distance algorithm is used to better find the spatial instances on the side of the vehicle body close to the camera position that are least affected by obstacles. The multi-dimensional (longitudinal priority instance trapezoidal boundary and transverse priority instance trapezoidal boundary) polygonal space (trapezoidal) boundary regression scheme with variable weights (α and β are transverse and longitudinal correction parameters respectively) supports the construction of horizontal and vertical parking spaces while also supporting the construction of spatial oblique parking spaces.

[0056] By setting the preset position, the instance area closest to the preset position is selected only through coordinates using a weighted distance algorithm, reducing the impact of three-dimensional obstacles within the surround image range on the instance segmentation of the parking space area, and the amount of calculation is small.

[0057] The method of the present invention can quickly realize spatial parking space detection without using a long-range ultrasonic radar by using a surround view image instance segmentation algorithm and optimizing the selection of image frames, and can greatly avoid the obstruction of parking spaces in the surround view image by three-dimensional obstacles in the surround view image, thereby more accurately detecting parking spaces and improving the accuracy of spatial parking space detection. In addition, by visually detecting spatial parking spaces, the problem that traditional ultrasonic radar cannot detect oblique parking spaces when detecting spatial parking spaces can be solved.

[0058] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, or two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0059] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A vision-based spatial parking space detection method, applied to electronic equipment of a vehicle, characterized in that: Receive the surround view image of the vehicle and perform brightness preprocessing on the surround view image; The surround view image after brightness preprocessing is input into the trained visual parking space instance segmentation model to obtain the first-level parking space instance area in the surround view image; For each of the first-level parking space instance regions, dilation and erosion operations are performed respectively, and then area screening is performed to remove instance regions whose pixel areas are smaller than the minimum pixel area of ​​theoretical parking spaces, and the remaining first-level parking space instance regions are used as second-level parking space instance regions; From the second-level parking space instance area, select an instance area closest to the preset position as the third-level parking space instance area; For the three-level parking space instance area, construct the maximum instance boundary of the horizontal space parking space; According to the required parking space size, a parking space is constructed based on the maximum instance boundary.

2. The vision-based spatial parking space detection method according to claim 1, characterized in that: The visual parking space instance segmentation model adopts an instance segmentation algorithm and is obtained by training the model using a surround-view parking space instance segmentation dataset; the surround-view parking space instance segmentation dataset is obtained by screening and annotating parking surround-view pictures; 3. The vision-based spatial parking space detection method according to claim 1, characterized in that: This method is triggered when a signal to start the parking space search function is received. If there is no parking space that meets the requirements, the search continues until a parking space is found or a signal to exit the parking space search function is received.

4. The vision-based spatial parking space detection method according to claim 1, characterized in that: The brightness preprocessing includes: Calculate the average brightness of the surround image; According to the average brightness and the preset normal brightness range, the brightness of the surround view image is corrected so that the average brightness of the surround view image is within the normal brightness range; Then the surround image is processed by median filtering.

5. The vision-based spatial parking space detection method according to claim 1, characterized in that: The surround parking space instance segmentation dataset is obtained in the following way: Collect a large number of parking surround view pictures, screen the parking surround view pictures, and retain the parking surround view pictures with at least one reasonable parking space to form a surround view parking space instance segmentation dataset; The images of the surround parking space instance segmentation dataset are annotated with parking space instances. The basic principle of annotation is to only annotate reasonable parking space instance areas, and there should be obstacles on at least one side of the parking space instance area.

6. The vision-based parking space detection method according to claim 1, characterized in that: The instance area closest to the preset position is: the instance area closest to the left and right surround view cameras of the vehicle and close to the middle of the surround view image.

7. The vision-based spatial parking space detection method according to claim 6, characterized in that: The three-level parking space example area is obtained by: Taking the center of the rear axle of the vehicle as the coordinate origin, the direction of the front of the vehicle as the y direction, and the direction perpendicular to the right side of the vehicle body as the x direction, calculate the center coordinate point of each secondary parking space instance area; The secondary parking space instance area is divided into the left and right sides of the vehicle body according to the central coordinate point, and the instance screening weighted distance L from the secondary parking space instance area on the left and right sides to the left and right surround view cameras is calculated respectively. l and L r : In the formula, x l ,y l is the x and y coordinates of the surround view camera on the left side of the vehicle body, x r ,y r is the x and y coordinates of the surround view camera on the right side of the vehicle body, x i ,y i are the x- and y-direction coordinates of each pixel point in a second-level parking space instance area, n is the total number of pixels in a second-level parking space instance area, and a is a preset value; Calculate L l and L r The minimum value of is obtained to obtain the three-level parking space instance area.

8. The vision-based spatial parking space detection method according to claim 1, characterized in that: The maximum instance boundary of the horizontal space parking space includes a longitudinal priority instance trapezoidal boundary and a transverse priority instance trapezoidal boundary, which are obtained by: The length of the bottom side of the trapezoid close to the vehicle is L1, and the length of the corresponding side of the trapezoid on the other end is L2. These two sides are parallel to each other, and the height of the trapezoid is H. (L1,L2) , the vertical priority instance boundary trapezoid construction formula is obtained as follows: Where W H is the corrected vertical priority instance boundary trapezoidal area, α is the vertical correction parameter; The formula for constructing the boundary trapezoid of the horizontal priority instance is: Where W L is the corrected lateral priority instance boundary trapezoidal area, β is the lateral correction parameter; W found respectively H and W L The maximum value is used to obtain the vertical priority instance trapezoidal boundary and the horizontal priority instance trapezoidal boundary.

9. The vision-based spatial parking space detection method according to claim 8, characterized in that: Based on the longitudinal priority trapezoidal boundary and the transverse priority trapezoidal boundary, the method for constructing a parking space is as follows: First, a standard vertical space parking space is constructed in the trapezoidal boundary of the longitudinal priority instance. If the conditions are met, a standard vertical space parking space is constructed. The size of the standard vertical space parking space is preset. Assuming that the waist of the trapezoidal boundary closer to the rear of the vehicle is the first side of the trapezoidal boundary, when the longitudinal priority instance trapezoidal boundary does not meet the construction of the standard vertical space parking space, calculate the angle between the first side of the longitudinal priority instance trapezoidal boundary and the bottom side of the trapezoidal boundary on the side close to the vehicle, and construct a standard size oblique parking space based on this angle and the first side of the trapezoidal boundary. If the space meets the requirements, construct a standard oblique space parking space; the size of the standard oblique space parking space is preset; If the construction of the standard vertical space parking space and the standard oblique space parking space is unsuccessful, then a standard horizontal space parking space is constructed in the trapezoidal boundary of the horizontal priority instance. The standard horizontal space parking space is parallel to the bottom edge of the trapezoidal boundary of the horizontal priority instance. If the space meets the requirements, then a standard horizontal space parking space is constructed. The size of the standard horizontal space parking space is preset. If the conditions for constructing a standard horizontal space parking space are still not met, there is currently no suitable parking space available.

10. A vision-based space parking detection system, characterized in that: The method comprises a memory, a processor and a program stored in the memory and executable on the processor, wherein the method according to any one of claims 1 to 9 is implemented when the processor executes the program.

11. A car, characterized in that: It includes a surround-view camera and the vision-based spatial parking space detection system as described in claim 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 9 are implemented.

Images