A method for detecting parking spaces in an indoor parking lot based on binocular vision

Abstract

The application provides a kind of indoor parking lot parking stall detection method based on binocular vision, comprising the following steps: S1. calibrating vehicle-mounted vision system;S2. utilize binocular camera to obtain indoor parking stall data set, and the data set of collected indoor parking stall is carried out data cleaning;S3. adopt marking software to mark the image in indoor parking stall data set;S4. construct parking stall identification model YOLOv4-MobileNetV3;S5. design light weight empty parking stall classification network;S6. design parking stall availability detection model;S7. the image with marking in step S3 is input into parking stall availability detection model, and parking stall detection result is obtained;S8. combined with target position information and final classification result, accurate effective parking stall detection result is comprehensively obtained.The application provides a kind of indoor parking lot parking stall detection method based on binocular vision, which can effectively reduce the interference of environmental factors, and facilitate the rapid and accurate identification of target parking stall.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automatic driving, and particularly relates to a parking space detection method for an indoor parking lot based on binocular vision. BACKGROUND

[0002] According to an investigation, the contradiction between the supply and demand of parking spaces and automobiles is increasingly prominent as the nationwide motor vehicle ownership approaches 4 billion. With the gradual reduction of parking space, the existing technology cannot effectively detect the parking environment, the difficulty of parking increases, and higher requirements are put forward for the parking-in ability of drivers. In order to better solve the parking problem, it is crucial to accurately detect the parking space information in the parking lot environment, judge whether it is a valid parking space, and timely prompt the existence of obstacles in the parking space.

[0003] The detection of parking spaces usually adopts a camera as a sensor to obtain parking spaces by recognizing the parking lines or parking angles on the ground, but requires the existence of complete straight lines or intersection points of line segments in the environment, and cannot completely extract some features such as light reflection and pollution. To solve such problems, deep learning technology is usually applied to parking line detection. Machine vision is mainly divided into monocular vision and binocular vision. Monocular vision images have distortion and poor anti-interference ability, and cannot directly obtain the depth information of the target. Binocular vision can effectively perceive the complex environment of parking lines and other planar targets, so the parking environment perception technology based on binocular vision will be more and more applied to automatic parking.

[0004] The following problems exist when using a binocular vision deep learning method to recognize parking lines: 1. When using machine vision to recognize parking space images, the effective parking space detection network needs to prepare parking space data in the parking process as the research basis, but the current public data set is a fish-eye data set and mainly detects parking angles, which is not suitable for this scenario; 2. The traditional parking space detection method has certain limitations when processing parking spaces in complex environments. In order to accurately identify the target parking space, a parking space detection model based on deep learning is introduced, but high-precision deep learning network models often have the problems of large parameters and slow speed; 3. When there are small obstacles such as roadblocks in the indoor parking space, it is often difficult to accurately detect them, which will affect the accuracy of parking space detection and even endanger life and property safety.

[0005] Therefore, there is an urgent need for a parking space detection method that is highly applicable, fast in detection rate, and can detect obstacles. SUMMARY

[0006] Therefore, the present application aims to provide a stereo vision-based indoor parking lot parking space detection method.

[0007] To achieve the above-mentioned purpose, the present application provides a stereo vision-based indoor parking lot parking space detection method, comprising the following steps:

[0008] S1. Calibrate the vehicle-mounted vision system;

[0009] S2. Obtain indoor parking space data set by using binocular camera, and clean the collected indoor parking space data set;

[0010] S3. Label the images in the indoor parking space data set by using labeling software, and store the labeling information in xml file;

[0011] S4. Construct parking space recognition model YOLOv4-MobileNetV3: 1). Replace the structure of the backbone network CPSDarknet53 in the neural network model YOLOv4 with MobileNetV3; 2). Delete the network structure of layers 17, 18, 19 and 20 in the backbone feature extraction network MobileNetV3; 3). Replace all ordinary convolutions of FPN+PAN with depth separable convolutions;

[0012] S5. Design lightweight empty parking space classification network: Conv_1 is traditional convolution, Block_1, 2, 3 is depth separable convolution;

[0013] S6. Design parking space availability detection model: learn from the two-stage algorithm idea, improve YOLOv4-MobileNetV3, that is, add the designed lightweight empty parking space classification network;

[0014] S7. Input the image with label in step S3 into the parking space availability detection model to obtain the parking space detection result, and the specific steps are as follows:

[0015] S7.1 Input the image with label in step S3 into YOLOv4-MobileNetV3 model to detect the parking space image and obtain the position information and the classification result corresponding to the image, which is the preliminary detection;

[0016] S7.2 Input the image after preliminary detection into the lightweight empty parking space classification network for secondary recognition, filter the preliminary detection result, and thus obtain the final classification result of whether the image corresponds to an effective parking space;

[0017] S8. Obtain accurate effective parking space detection result by combining target position information and final classification result.

[0018] Further, in the step S1, the internal parameters of the binocular camera are calibrated by using a calibration board, and distortion calibration is performed.

[0019] Further, in the step S2, the indoor parking space dataset is the image information of the parking space collected by the binocular camera moving with the vehicle during the parking process.

[0020] Further, in the step S3, the labeled content includes the parking space angle, the effective parking space, and the end region.

[0021] Further, the process of secondary recognition of the lightweight empty parking space classification network is as follows:

[0022] 1). Input the image after preliminary detection, and Conv_1 extracts the preliminary edge features of the image in detail.

[0023] 2). Block_1, 2, and 3 further extract the detailed features of the image in depth.

[0024] 3). After extracting deep features through three layers of depth separable convolution layers, the image is sent to a fully connected neural network for classification.

[0025] Further, the identity mapping structure is added in the depth separable convolution layers Block_1, 2, and 3.

[0026] Further, the loss function of the lightweight empty parking space classification network is selected as the cross-entropy error.

[0027] Further, in the lightweight empty parking space classification network, the activation function of Conv_1 is selected as Leaky ReLU, and the activation function of the feature extraction other than Conv_1 is selected as ReLU6.

[0028] The beneficial effects of the present application are as follows:

[0029] 1. The indoor parking lot parking space detection method based on binocular vision can observe the parking space image of the indoor environment at different orientations through binocular vision during the parking process, and determine the parking space feature region, solving the problem that the existing parking space detection method is not suitable for indoor parking lot scenes.

[0030] 2. The lightweight YOLOv4-MobileNetV3 effective parking space detection model is designed, which can detect and identify the indoor parking space, and has faster detection rate and higher accuracy.

[0031] 3. The application adds a lightweight empty parking space classification network on the basis of the YOLOv4-MobileNetV3 effective parking space detection model for fusion judgment, removes the situation where obstacles exist but are judged as effective parking spaces, and timely prompts, which can improve the accuracy of the model while ensuring small parameter improvement.

[0032] Other advantages, objects, and features of the present application will be apparent to those skilled in the art upon reading the following specification, and will be more readily apparent when the following specification is taken in conjunction with the accompanying drawings. The objects and other advantages of the present application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 A flowchart of a parking space detection method based on binocular vision for an indoor parking lot according to the present application;

[0034] Figure 2 For the parking process;

[0035] Figure 3 For the parking space feature region annotation map;

[0036] Figure 4 YOLOv4-MobileNetV3 network structure;

[0037] Figure 5 Empty parking space classification network structure;

[0038] Figure 6 Improved YOLOv4-MobileNetV3 network structure;

[0039] Figure 7a Detection result of obstacle one;

[0040] Figure 7b Detection result of obstacle two;

[0041] Figure 7c Detection result of obstacle three;

[0042] Figure 7d Detection result of empty parking space one;

[0043] Figure 7e Detection result of empty parking space two;

[0044] Figure 7f Detection result of empty parking space three. DETAILED DESCRIPTION

[0045] In order to make the technical solutions, advantages and objectives of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely below in conjunction with the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without any creative effort fall within the protection scope of the present application.

[0046] The structure of the YOLOv4 network: divided into three parts of Backbone, Neck and Head. The Backbone main part is mainly composed of CSPDarknet53, and CSPDarkNet53 is a feature extraction network used for preliminary feature extraction of images for subsequent structures; the Neck is mainly composed of a spatial pyramid pooling structure (SPP) and FPN+PAN, which combines the information of different feature layers to realize the reservation of features and extract more detailed information to improve the network feature fusion capability; the Head is used to predict the output and compare with the real label to calculate the loss function to train the network parameters.

[0047] The characteristics of MobileNet are: a lightweight classification network for mobile terminal and embedded device design, which has smaller size and less calculation amount compared with traditional convolutional neural network, but has higher precision.

[0048] As shown in Figure 1 The present application provides a kind of indoor parking lot parking space detection method based on binocular vision, comprising the following steps:

[0049] S1. calibrating vehicle-mounted vision system: using calibration board to calibrate the internal parameters of binocular camera, and carrying out distortion calibration.

[0050] S2. using binocular camera to obtain indoor parking space dataset, and carrying out data cleaning to the collected indoor parking space dataset, and eliminating the sample with poor quality; indoor parking space dataset is the image information of parking space collected by binocular camera moving with vehicle during parking process.

[0051] S3. using annotation software to annotate the image in indoor parking space dataset, the annotation content includes parking space angle, effective parking space and terminal area, and the annotation information is stored in xml file.

[0052] The annotation content, i.e. parking space feature region selection. In actual parking process, vehicle needs to approach parking space by continuous movement, and the visible angle of parking space relative to vehicle also changes continuously. The process when vehicle is close to parking line is divided into a, b and c steps:

[0053] 1). First, drive the car to the target parking space, at this time only the parking angle of the parking space can be seen in the view, cut the car back into the parking space at about 45°, the whole view of the parking space at 45° can be observed in this step;

[0054] 2). When approaching the parking space, turn the steering wheel in the opposite direction to make the car about 30° with the parking space;

[0055] 3). Finally, slowly straighten the front wheels and drive into the parking space, at this time the view is 0° and only the parking end area can be observed to assist parking.

[0056] When the distance is far away, only step 1 is needed, as shown in Figure 2 .

[0057] In the actual parking process, only the parking angle can be observed in the first half of step 1, and the whole parking space is used to determine whether it is a valid parking space when starting to drive into the parking space. When it is detected that it is a valid parking space, steps 2 and 3 are continued, and when gradually parking into the parking space, the whole view of the parking space cannot be captured, and the end area of the parking space is detected to assist parking, so the parts that need to be marked include the parking angle, the valid parking space and the end area, as shown in Figure 3 .

[0058] S4. Construct a parking space recognition model YOLOv4-MobileNetV3, including the following operations:

[0059] 1). Replace the structure of the backbone network CPSDarknet53 in the neural network model YOLOv4 with MobileNetV3 to reduce the network computation and parameter amount;

[0060] 2). Delete the network structure of layers 17, 18, 19 and 20 in the backbone feature extraction network MobileNetV3;

[0061] When the input size is 416*416*3, select three scales of feature layers 52*52*40, 26*26*112 and 13*13*160 to send to the Neck structure for feature fusion, 52*52 feature layer for small target prediction, 26*26 for medium target prediction, and 13*13 for large target prediction, and the feature vector size of the output of the network can be obtained. 52*52*24, 26*26*24 and 13*13*24.

[0062] 3). Replace all ordinary convolutions of FPN+PAN with depth separable convolutions to achieve lightweight purpose, greatly reducing the computation and parameter amount of ordinary convolutions, and the finally constructed YOLOv4-MobileNetV3 network structure is as shown in Figure 4 .

[0063] S5. Design a lightweight empty parking space classification network, such as Figure 5 As shown in the figure, Conv_1 is a traditional convolution, Block_1, 2, 3 is a depth separable convolution; an identity mapping structure is added in the depth separable convolution layer Block_1, 2, 3.

[0064] The loss function of the lightweight empty parking space classification network is selected as cross-entropy error. In the lightweight empty parking space classification network, the activation function of Conv_1 is selected as Leaky ReLU, and the activation function of the feature extraction other than Conv_1 is selected as ReLU6.

[0065] S6. Design a parking space availability detection model: learn from the two-stage algorithm idea, improve YOLOv4-MobileNetV3, that is, add the designed lightweight empty parking space classification network.

[0066] S7. Input the image with label in step S3 into the parking space availability detection model to obtain the parking space detection result, and the specific steps are as follows:

[0067] S7.1 Input the image with label in step S3 into the YOLOv4-MobileNetV3 model to detect the parking space image and obtain the position information and the classification result corresponding to the image, which is the preliminary detection, wherein the parking space image without obstacles is marked as free, and the parking space image with obstacles is marked as occupied.

[0068] S7.2 Input the image after preliminary detection into the lightweight empty parking space classification network for secondary identification, and screen the preliminary detection result to obtain the final classification result of whether the parking space corresponding to the image is a valid parking space.

[0069] The process of secondary identification of the lightweight empty parking space classification network is as follows:

[0070] 1). Input the image after preliminary detection, and Conv_1 extracts the preliminary edge features of the image;

[0071] 2). Block_1, 2, 3 further extracts the detailed features of the image in depth;

[0072] 3). After extracting the deep features through the three layers of depth separable convolution layers, send them to the fully connected neural network for classification.

[0073] The improved YOLOv4-MobileNetV3 model is as shown in the figure. Figure 6 The improved YOLOv4-MobileNetV3 model can improve the speed and reduce the network parameters under the premise of ensuring the accuracy.

[0074] S8. The target position information and the final classification result are combined to obtain an accurate effective parking space detection result.

[0075] Detection model comparison and analysis

[0076] The performance of the improved YOLOv4-MobileNetV3 model in detecting effective parking spaces in an indoor parking lot is compared with that of common target detection models, and the test set is tested respectively, and the final detection result is shown in Table 1.

[0077] Table 1: Accuracy of each model

[0078]

[0079] The accuracy of the improved YOLOv4-MobileNetV3 model is improved by 3.58% compared with the original model, and it can effectively identify small obstacles such as roadblocks on the parking space.

[0080] Detection experiment

[0081] Figures 7a-7d The detection result graph obtained by using the method of the present application is shown. Figure 7c It can be seen that when there is a small obstacle on the parking space, the improved YOLOv4-MobileNetV3 model can recognize it as a non-empty state, and has good generalization ability. Figure 7e It can be seen that even in the case of strong light and interference of the parking space, the improved YOLOv4-MobileNetV3 model can effectively identify the empty parking space, has robustness, and the improvement effect meets the scene requirements and has strong applicability.

[0082] Finally, it should be pointed out that the above examples are only used to illustrate the technical solutions of the present application and are not limiting, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, and they should be included in the protection scope of the present application.

Claims

1. A method for detecting parking spaces in an indoor parking lot based on binocular vision, characterized in that, The method comprises the following steps: S1. Calibrating the vehicle-mounted vision system; S2. Using a binocular camera to obtain an indoor parking space dataset, and performing data cleaning on the collected indoor parking space dataset; S3. Using a labeling software to label the images in the indoor parking space dataset, and storing the labeling information in an xml file; The labeling content includes a parking space angle, an effective parking space, and an end region; S4. Constructing a parking space recognition model YOLOv4-MobileNetV3: 1). Replacing the structure of the backbone network CPSDarknet53 in the neural network model YOLOv4 with MobileNetV3; 2). Deleting the network structures of layers 17, 18, 19, and 20 in the backbone feature extraction network MobileNetV3; 3). Replacing all ordinary convolutions of FPN+PAN with depth separable convolutions; S5. Designing a lightweight empty parking space classification network: Conv_1 is a traditional convolution, and Block_1, 2, and 3 are depth separable convolutions; The depth separable convolution layers Block_1, 2, and 3 are added with an identity mapping structure; The loss function of the lightweight empty parking space classification network is selected as a cross-entropy error; In the lightweight empty parking space classification network, the activation function of Conv_1 is selected as LeakyReLU, and the activation function of the feature extraction other than Conv_1 is selected as ReLU6; S6. Designing a parking space availability detection model: connecting the YOLOv4-MobileNetV3 model and the lightweight empty parking space classification network in series to form a two-stage detection network; S7. Inputting the images with labels in step S3 into the parking space availability detection model to obtain parking space detection results, and the specific steps are as follows: S7.

1. Inputting the images with labels in step S3 into the YOLOv4-MobileNetV3 model to detect the parking space images, obtaining position information and classification results corresponding to the images, which is a preliminary detection; S7.

2. Inputting the images after the preliminary detection into the lightweight empty parking space classification network for secondary recognition, screening the results of the preliminary detection, and thus obtaining the final classification results of whether the images correspond to effective parking spaces; The secondary recognition process of the lightweight empty parking space classification network is as follows: 1). Inputting the images after the preliminary detection, and using Conv_1 to finely extract the preliminary edge features of the images; 2). Using Block_1, 2, and 3 to further extract the detailed features of the images in depth; 3). After extracting the deep features through the three depth separable convolution layers, inputting the deep features into a fully connected neural network for classification; S8. Combining the target position information and the final classification results to comprehensively obtain accurate effective parking space detection results.

2. The indoor parking space detection method based on binocular vision according to claim 1, characterized in that: In step S1, the internal parameters of the binocular camera are calibrated by using a calibration board, and distortion calibration is performed.

3. The indoor parking space detection method based on binocular vision according to claim 1, characterized in that: In step S2, the indoor parking space dataset is image information of parking spaces collected by the binocular camera moving with the vehicle during the parking process.

Images