A method for identifying the status of parking spaces based on an improved YOLO model

Abstract

The present invention discloses a parking space status recognition method based on an improved YOLO model, including: constructing a parking space detection data set and dividing it into a training set and a test set; constructing an improved YOLOV4-tiny network model, wherein the original backbone network CSPDarknet53-tiny of the YOLOV4-tiny network model is replaced with a lightweight network mobilenetv3, an SE attention mechanism is added before the feature pyramid FPN, the Mish activation function is replaced with the original Leaky ReLU activation function, and the EiOU loss function is replaced with the original CiOU loss function. The SE attention mechanism includes a squeezing module, an excitation module and a fusion module; training the improved YOLOV4-tiny network model with the training set; inputting the image to be tested into the trained improved YOLOV4-tiny network model to obtain the parking space status recognition result. The present invention can improve the detection speed on the premise of accurately recognizing the parking space status.

Description

Technical Field

[0001] The present invention relates to the fields of image processing, object detection, deep learning, etc., and particularly relates to a method for identifying the state of a parking space based on an improved YOLO model. Background Art

[0002] In recent years, with the rapid growth of the automotive industry, assisted by artificial intelligence and big data, driverless technology has made remarkable progress. As an important part of driverless technology, the research and popularization of automatic parking can effectively solve the problems of road traffic safety and parking difficulties. And as the most important part of automatic parking, environmental perception is the primary prerequisite for executing the entire automatic parking process. Identifying the state of the parking spaces around a vehicle is an important part of the environmental perception process and is also a prerequisite for accurately identifying parking spaces and precisely parking a vehicle into a parking space. Identifying the state of a parking space can play an important auxiliary role in the process of people's daily parking into an empty parking space. Therefore, it is of great significance to quickly and accurately identify the state of a parking space.

[0003] In recent years, significant breakthroughs have also been made in the algorithms of object detection technology. Nowadays, object detection methods based on convolutional neural networks have surpassed traditional object detection methods and become the mainstream methods of current object detection. Currently, the mainstream object detection algorithms based on convolutional neural networks are mainly divided into two categories. The first category is two-stage object detection algorithms, represented by early R-CNN, Fast R-CNN, etc. This algorithm first generates candidate regions (region proposals), and then uses a convolutional neural network to classify the candidate regions (and also correct the positions). Because multiple detection and classification processes are required, this method has high accuracy but relatively slow speed. The other category is one-stage object detection algorithms, represented by the YOLO series (You Only Look Once), SSD (Single Shot multibox Detector), OverFeat, RetinaNet, etc. Such algorithms are end-to-end detection algorithms that do not require the stage of generating candidate regions and directly generate the class probabilities and position coordinate values of objects. The final detection results can be directly obtained through a single detection. Therefore, they have a faster detection speed compared with the two-stage algorithms.

[0004] With the continuous development of the YOLO series of algorithms, in order to solve the problem of excessive number of parameters and computational complexity of convolutional neural network models, a lightweight model, the YOLOV4-tiny network, was proposed in 2020. The overall network structure has 38 layers, uses three residual units, and the activation function is LeakyReLU. The classification and regression of targets are changed to use two feature layers, and the Feature Pyramid Network (FPN) is used when merging effective feature layers. It has significant performance advantages compared to other versions of lightweight models. However, the recognition accuracy of the ordinary YOLOV4-tiny object detection algorithm is affected by factors such as natural weather and light intensity, and it cannot guarantee the real-time performance and accuracy of detection in different scenarios. Summary of the Invention

[0005] In order to be able to identify the parking space status in real time and accurately, the present invention provides a method for identifying the parking space status based on an improved YOLO model, which can ensure a high speed and accuracy of parking space status recognition, has good stability and strong robustness, and can be applied to an automatic parking system.

[0006] In order to achieve the object of the present invention, a method for identifying the parking space status based on an improved YOLO model provided by the present invention includes the following steps:

[0007] Construct a parking space detection data set and divide it into a training set and a test set;

[0008] Construct an improved YOLOV4-tiny network model. Among them, the original backbone network CSPDarknet53-tiny of the YOLOV4-tiny network model is replaced with a lightweight network mobilenetv3, an SE attention mechanism is added before the Feature Pyramid Network (FPN), the Mish activation function is used to replace the original Leaky ReLU activation function, and the EiOU loss function is used to replace the original CiOU loss function. The SE attention mechanism includes a squeezing module, an excitation module and a fusion module. The squeezing module is used to perform global average pooling on the feature map u c to obtain a compressed value z through global average pooling c The excitation module is used to process the value z c to obtain the channel weight value s of each channel. The fusion module is used to assign weights to the feature map u using the channel weight value s of each channel c to obtain the feature map X;

[0009] Use the training set to train the improved YOLOV4-tiny network model;

[0010] Input the image to be tested into the trained improved YOLOV4-tiny network model to obtain the recognition result of the parking space status.

[0011] Furthermore, constructing the parking space detection dataset and dividing it into a training set and a test set includes:

[0012] First, collect a part of parking space images by real vehicle to construct a dataset. The images in the dataset cover as many parking scenarios as possible, and integrate the constructed dataset with the open-source dataset;

[0013] Annotate the integrated dataset, including free parking spaces and occupied spaces, and divide it into a training set and a test set;

[0014] Enhance the data.

[0015] Furthermore, in the squeezing module, the value of z c is calculated as follows:

[0016]

[0017] In the formula, W represents the width of the feature map, H represents the height of the feature map, C represents the number of channels of the feature map, c represents the feature map after convolution.

[0018] Furthermore, the excitation module includes two fully connected layers.

[0019] Furthermore, each channel weight value s generates a vector S. The fusion module multiplies the H*W values of each channel in the feature map u c by the weight value of the corresponding channel in S, and finally obtains the feature map X. The size of the feature map X is exactly the same as that of the feature map u c exactly the same.

[0020] Furthermore, the calculation formula of the Mish activation function is as follows:

[0021] Mish = x * tanh(ln(1 + x ))

[0022] In the formula, x represents the input of the activation function.

[0023] Furthermore, the EiOU loss function includes overlap loss, center distance loss, and width-height loss.

[0024] Furthermore, the penalty term L of the EiOU loss function EIOU The formula is

[0025]

[0026] In the formula, IOU represents the intersection over union, b represents the center point of the prediction box, b gtrepresents the center point of the ground truth box, ρ represents the Euclidean distance between the two center points, n represents the diagonal distance of the smallest enclosing region that simultaneously contains the predicted box and the ground truth box, C w and C h represent the width and height of the smallest bounding box covering the two Boxes. L EIOU represents the total loss, L IOU represents the overlap loss, L dis represents the center distance loss, L asp represents the width-height loss, w represents the width of the predicted box, w gt represents the width of the ground truth box, h represents the height of the predicted box, h gt represents the height of the ground truth box.

[0027] Furthermore, the improvement of the YOLOV4-tiny network model also includes: replacing the 3*3 convolution in the original Yolo Head with a depthwise separable convolution.

[0028] Furthermore, the depthwise separable convolution includes a depthwise convolution and a pointwise convolution.

[0029] Compared with the prior art, the beneficial effects of the present invention are at least as follows: [[ID=2,7]]

[0030] The present invention conducts parking space status detection based on the improved YOLO4-tiny, combines the recognition of parking space status with deep learning, can provide an effective method for real-time detection of parking space status recognition, and can well detect the vacant parking spaces and occupied parking spaces in the image.

[0031] The present invention adopts the method of combining the attention mechanism SENet with the YOLOV4-tiny network, and simultaneously adopts the Mish activation function and the EiOU LOSS loss function, which can improve the network feature extraction ability while ensuring a high detection accuracy.

[0032] The present invention adopts the lightweight neural network mobilenetv3 and depthwise separable convolution, significantly reducing the number of network parameters, which can improve the detection speed, has good real-time performance, and can reduce the operation cost of the object detection algorithm model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flowchart of a method for recognizing parking space status based on an improved YOLO model provided by an embodiment of the present invention.

[0034] Figure 2 is a schematic diagram of Mosaic data augmentation in an embodiment of the present invention.

[0035] Figure 3 is a schematic diagram of the YOLOV4-tiny network structure.

[0036] Figure 4 It is the flowchart of the SE attention mechanism algorithm in the embodiment of the present invention.

[0037] Figure 5 It is the schematic diagram of depthwise separable convolution in the embodiment of the present invention.

[0038] Figure 6 It is the schematic diagram of the test result in the embodiment of the present invention. Specific embodiments

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in conjunction with the present application.

[0040] Please refer to Figure 1 , a parking space status recognition method based on an improved YOLO model provided by the present invention includes the following steps:

[0041] Step 1: Construct a parking space detection dataset and perform preprocessing.

[0042] Step 1 specifically includes the following sub-steps:

[0043] Step 1.1: First, collect a part of the dataset of parking spaces by real vehicle, covering as many parking scenarios as possible (such as covering brick floors and grasslands, indoor and outdoor, direct sunlight and night lights, etc. scenarios), and after removing invalid pictures, integrate the dataset collected by real vehicle and the open-source ps2.0 dataset together.

[0044] Among them, the ps2.0 dataset is the most typical open-source dataset of parking spaces in the academic community, which was publicly released by Tongji University in 2018.

[0045] Step 1.2: Label the images in the dataset and divide the dataset into a training set and a test set.

[0046] In some embodiments of the present invention, the labelimg software is used to label the images, and the integrated dataset is labeled in the VOC data format. The dataset is mainly labeled into two categories: vacant slot and occupied slot, and then the dataset is divided into a training set and a test set.

[0047] Step 1.3: Perform data augmentation on the data.

[0048] In some embodiments of the present invention, the Mosaic method is used to augment the data. Please refer to Figure 2, randomly select 4 pictures from the dataset, scale, rotate, and arrange them to form a new picture. This not only greatly increases the number of pictures input into the network, but also speeds up the training process, achieving the effect of data augmentation.

[0049] Step 2: Construct an improved YOLO network model.

[0050] Compare various current YOLO models and decide to select YOLOV4-tiny as the model for object detection to be improved. Because it is a lightweight model, with only 6 million parameters, which is one-tenth of YOLOV4, this greatly improves the detection speed. At the same time, it uses a Feature Pyramid Network (FPN) during the process of merging effective feature layers, ensuring the detection accuracy to a certain extent. The existing original structure diagram of YOLOV4-tiny is as Figure 3 shown, including three parts: the backbone feature extraction network CSPDarknet53-tiny, the Feature Pyramid Network (FPN), and Yolo Head. YOLOV4-tiny has the characteristics of multi-task, end-to-end, attention mechanism, and multi-scale: multi-task means simultaneously completing the classification and regression of the target, realizing parameter sharing, and avoiding overfitting; end-to-end means that the model directly gives the prediction information of classification and regression after receiving image data; the attention mechanism focuses on the features of the target area and processes them in detail, thereby improving the processing speed; the multi-scale characteristic is to fuse the data after downsampling and upsampling, and its function is to be able to segment targets of various scale sizes.

[0051] The improvement of the YOLOV4-tiny network model in the present invention includes:

[0052] 1. Introduce the lightweight network mobilenetv3 to replace the original backbone feature network CSPDarknet53-tiny. The mobilenet network is a lightweight convolutional neural network proposed by the Google team and focuses on embedded devices. The main advantages of mobilenetv3 are as follows: it uses a large number of 1*1 and 3*3 convolutions instead of 5*5 convolutions, reducing the number of parameters; first uses a 1*1 convolution and then a 3*3 convolution, retaining the high-dimensional feature space and reducing the delay of backpropagation; it introduces residual blocks and lightweight attention mechanisms and is inspired into a bottleneck structure, which can better extract features; it uses two AutoML techniques. First, use MnasNet to search for a rough structure, then use reinforcement learning to select the optimal configuration from a set of discrete choices, and then use NetAdapt to fine-tune the architecture, which can adjust the under-utilized activation channels with a small reduction. Replacing the backbone feature extraction network with the lighter mobilenetv3 can achieve the lightweight of the network model and balance speed and accuracy.

[0053] 2. After generating the feature map u through the backbone feature extraction network c , an SE attention mechanism is added before the Feature Pyramid Network (FPN), as Figure 4 shown. The SE attention mechanism mainly includes three modules:

[0054] (1) Squeeze (squeezing operation): Perform global average pooling on the feature map u after convolution to generate a 1*1*C vector, and finally compress it into a numerical value z c . Its calculation formula is as follows: c .

[0055]

[0056] In the formula, W represents the width of the feature map, H represents the height of the feature map, C represents the number of channels of the feature map, c represents the feature map after convolution, i represents the i-th unit of the height, and j represents the j-th unit of the width.

[0057] (2) Excitation (excitation operation): Process the numerical value z obtained through the squeezing operation through two fully connected layers to obtain the channel weight value s. After passing through two fully connected layers, different numerical values in the channel weight value s represent the weight information of different channels. c (3) Scale (fusion operation): Assign weights to the feature map u for each channel weight value s. Multiply the vector S generated by each channel weight value s with the corresponding channels of the feature map u

[0058] , that is, multiply the H*W numerical values of each channel in the feature map u c by the weight value of the corresponding channel in S. Finally, the feature map X is obtained, and the size of the feature map X is exactly the same as that of the feature map u c . c c .

[0059] Based on extracting features from the feature map, the SE attention mechanism adaptively assigns channel weights, making the feature maps with greater effects have a greater impact on the results. Therefore, it is more effective in extracting features than ordinary convolutional layers.

[0060] 3. Replace the existing Leaky ReLU activation function with the Mish activation function. The calculation formula of the Mish activation function is as follows:

[0061] Mish = x * tanh(ln(1 + x ))

[0062] In the formula, x represents the input of the activation function.

[0063] ​The Mish activation function is a non-monotonic activation function that helps maintain a relatively small negative value. Secondly, it is a smooth function that allows better information to penetrate into the neural network, contributing to improved generalization ability. It has good gradient descent performance, and the function's unboundedness avoids the saturation problem.

[0064] 4. At the output end, EiOU LOSS is used instead of the previous CiOU LOSS as the final loss function. The penalty term of EiOU separates the influence factor of the aspect ratio based on the penalty term of CiOU and calculates the lengths and widths of the target box and the anchor box separately. This loss function consists of three parts: overlap loss, center distance loss, and width-height loss. The first two parts follow the method in CiOU, but the width-height loss directly minimizes the difference between the width and height of the target box and the anchor box, resulting in a faster convergence speed. The penalty term formula is as follows:

[0065]

[0066] In the formula, IOU represents the intersection over union, b represents the center point of the predicted box, b gt represents the center point of the ground truth box, ρ represents the Euclidean distance between the two center points, n represents the diagonal distance of the smallest enclosing region that contains both the predicted box and the ground truth box, C w and C h represent the width and height of the smallest bounding box covering the two boxes. L EIOU represents the total loss, L IOU represents the overlap loss, L dis represents the center distance loss, L asp represents the width-height loss, w represents the width of the predicted box, w gt represents the width of the ground truth box, h represents the height of the predicted box, h gt represents the height of the ground truth box.

[0067] 5. Replace the 3×3 convolution in the Yolo Head with depthwise separable convolution. Depthwise separable convolution mainly consists of two processes, which are combined by depthwise convolution and pointwise convolution to extract feature maps. One convolution kernel in depthwise convolution is responsible for one channel, and only one convolution kernel convolves one channel. The number of channels of the feature map generated in this process is the same as that of the input, and the number of feature maps after depthwise convolution is the same as the number of channels of the input layer, unable to expand the feature map. Moreover, this operation performs convolution on each channel of the input layer independently and does not effectively utilize the feature information of different channels at the same spatial position. Therefore, pointwise convolution is needed to combine these feature maps to generate new feature maps. Compared with conventional convolution operations, the number of parameters and computational cost of depthwise separable convolution are significantly reduced.

[0068] Step 3: Determine the learning rate, batch size, and epoch, input the training set images into the improved network model for training, and obtain the trained improved YOLOV4-tiny network model.

[0069] In some embodiments of the present invention, the input size of the image is set to 416×416, and the initial parameters are set as follows: the learning rate is set to 0.001, the batch size is set to 40 due to the limitation of the video card memory, and the number of epoch iterations is set to 500. After the initial 100 epochs, the learning rate is decayed to 0.1 times the original value every 200 epoch iterations. The learning rate is initially set to a relatively large value of 0.001 to make the descent speed faster at the beginning of training, and the subsequent gradual decay of the learning rate is to enable the model to converge as soon as possible. After multiple iterations, the loss value will tend to be stable, and the network parameter model at this time is saved.

[0070] Step 4: Input the image to be detected into the trained improved YOLOV4-tiny network model for detection and recognition to obtain the recognition result of the parking space status.

[0071] In some embodiments of the present invention, the improved YOLOV4-tiny network model is used for detection. Figure 6 The following shows some recognition effects. Figure (a) shows that an empty parking space is recognized, and Figure (b) shows that an occupied parking space is recognized.

[0072] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, rather than limitations on the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made on the basis of the above description. It is not necessary and impossible to enumerate all the implementation manners here. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the claims of the present invention.

Claims

1. A method for identifying the parking space status based on an improved YOLO model, characterized in that, It includes the following steps: Construct a parking space detection dataset and divide it into a training set and a test set; Build an improved YOLOV4-tiny network model. Among them, replace the original backbone network CSPDarknet53-tiny of the YOLOV4-tiny network model with the lightweight network mobilenetv3, add the SE attention mechanism before the feature pyramid FPN, replace the original Leaky ReLU activation function with the Mish activation function, and replace the original CiOU loss function with the EiOU loss function. The SE attention mechanism includes a squeezing module, an excitation module, and a fusion module. The squeezing module is used to perform global average pooling on the feature map u c and compress it to obtain a value z c . The excitation module is used to process the value z c to obtain the channel weight value s of each channel. The fusion module is used to assign weights to the feature map u c using the channel weight value s of each channel to obtain the feature map X; Use the training set to train the improved YOLOV4-tiny network model; Input the image to be detected into the trained improved YOLOV4-tiny network model to obtain the recognition result of the parking space state.

2. The method for identifying the parking space status based on the improved YOLO model according to claim 1, wherein The construction of the parking space detection dataset and the division into a training set and a test set include: First, collect a part of parking space images by real vehicle to construct a dataset. The images in the dataset cover as many parking scenarios as possible, and integrate the constructed dataset with the open-source dataset; Annotate the integrated dataset, including free parking spaces and occupied spaces, and divide it into a training set and a test set; Enhance the data.

3. The parking space status recognition method based on the improved YOLO model according to claim 1, characterized in that, In the extrusion module, the value z c is calculated as follows: Wherein, W represents the width of the feature map, H represents the height of the feature map, C represents the number of channels of the feature map, and u c represents the feature map after convolution, i represents the i-th unit of the height, and j represents the j-th unit of the width.

4. The parking space status recognition method based on an improved YOLO model according to claim 1, characterized in that, The excitation module includes two layers of fully connected layers.

5. The method for identifying the parking space status based on the improved YOLO model according to claim 1, wherein Each channel weight value s generates a vector S, and the fusion module multiplies the H*W values of each channel in the feature map u c by the weight value of the corresponding channel in S, and finally obtains a feature map X, and the size of the feature map X is the same as that of the feature map u c exactly the same.

6. The method for identifying the parking space status based on an improved YOLO model according to claim 1, wherein The calculation formula of the Mish activation function is as follows: Mish=x*tanh(ln(1+e x )) In the formula, x represents the input of the activation function.

7. A method for identifying the parking space status based on an improved YOLO model according to claim 1, characterized in that, The EiOU loss function includes overlap loss, center distance loss and width-height loss.

8. A method for identifying the parking space status based on an improved YOLO model according to claim 1, characterized in that, The penalty term L of the EiOU loss function EIOU The formula is Where IOU represents the intersection over union, b represents the center point of the predicted bounding box, and b gt represents the center point of the ground truth bounding box, ρ represents the Euclidean distance between the two center points, n represents the diagonal distance of the smallest enclosing region that contains both the predicted bounding box and the ground truth bounding box, C w and C h represent the width and height of the smallest bounding box that covers the two boxes. L EIOU represents the total loss, L IOU represents the overlap loss, L dis represents the center distance loss, L asp represents the width-height loss, w represents the width of the predicted bounding box, and w gt represents the width of the ground truth bounding box, h represents the height of the predicted bounding box, and h gt represents the height of the ground truth bounding box.

9. A method for identifying the status of a parking space based on an improved YOLO model according to any one of claims 1-8, characterized in that, The improvement of the YOLOV4-tiny network model also includes: replacing the 3*3 convolution in the original Yolo Head with a depthwise separable convolution.

10. The method for identifying the parking space status based on the improved YOLO model according to claim 9, wherein, The depthwise separable convolution includes a depthwise convolution and a pointwise convolution.

Images